- Open Access
Recognition and localization of actinidia arguta based on image recognition
© The Author(s). 2019
- Received: 29 October 2018
- Accepted: 9 January 2019
- Published: 25 January 2019
In the process of picking, actinidia arguta has difficulty in image recognition and occlusion problems, and there are few studies on kiwifruit recognition. Based on this, this study uses the color model to perform image basic processing and uses frequency domain enhancement to process the image. Simultaneously, in the frequency domain of the image, this study applied filtering to the original image of kiwifruit orchard, used homomorphic filtering to enhance the image of actinidia arguta orchard, highlighted the characteristics of actinidia arguta trunk, and reduced the influence of background noise on the recognition of actinidia arguta trunk. In addition, this study used a binocular stereo vision system for fruit location recognition to improve recognition accuracy. Finally, the effectiveness of the research method is verified by experimental research. The results show that the proposed algorithm performs well and can provide theoretical references for subsequent related research.
- Image recognition
- Fruit recognition
Kiwifruit needs intensive picking during the maturity period, and picking is one of the most time-consuming steps in the kiwifruit planting process, which is currently done manually. With the large number of peasants working in cities and the aging of the population, the rural labor resources are becoming increasingly tense, and the demand for agricultural machinery operations is becoming more and more urgent. In order to make the kiwi picking robot reach a practical level, it is the key to improve its picking efficiency .
In the 1960s, American scholars Schertz and Brown first proposed the use of robots to pick fruits. Since then, various fruit and vegetable picking robots have been extensively studied and have achieved a lot of results . In 2004, Zhang Tiezhong used a back-propagation (BP) neural network-based algorithm to segment mature strawberry fruit from the background. The algorithm converts the acquired image into the hue, saturation, value (HSV) color space and uses the value of the 33 neighborhood pixel H channel as the input of the 3-layer BP neural network. The algorithm is used to train 20 images of 400 or 300 pixels. After 100 cycles, the experimental error is about 0.001, which indicates that the obtained network weight can segment the image well . In 2010, Lu Qiang et al. proposed a method for identifying citrus in natural scenes. This method divides citrus images by the maximum inter-class variance method based on G-B color difference components. If the fruit overlaps, the watershed segmentation algorithm based on distance transformation is used to segment. If the leaf occlusion occurs, the minimum convex hull algorithm is used to reduce the influence of occlusion on the recognition. Finally, the least squares fitting is performed on the segmented image to determine the center and radius of the citrus fruit to realize the recognition of citrus. It can be concluded from a series of experiments that the recognition accuracy of citrus recognition using this algorithm is 2.87% . In 2015, Zhao De’an et al. of Jiangsu University proposed an algorithm combining the improved R-G color difference segmentation algorithm and the quadratic segmentation method to identify night kiwifruit images. The experimental results show that the correct recognition rate of the algorithm for a single unobstructed kiwifruit is 83.7% .
Binocular vision is an important part of computer vision. According to binocular stereo vision technology, researchers can obtain the stereo information of an object and calculate the three-dimensional coordinates of the object to achieve the positioning of the object . Binocular vision technology originated in 1977, and Professor Marr proposed the computer binocular vision theory in the USA. Since then, the binocular vision theory has been continuously improved, and its application in the field of fruit and vegetable picking robots has been continuously matured . In 2002, Professor Takahashi of Hirosaki University in Japan used binocular stereo vision technology to locate kiwifruit in the orchard. First, the three-dimensional space is divided into several cross-sections according to the line-of-sight distance, and then the three-dimensional information of the object is reconstructed by integrating the two-dimensional images of the cross-sections. It has been proved by experiments that the error of measuring the depth distance using this method is controlled within ± 5%. When there are 20~30 fruits in the image, the discrimination rate of fruit is greater than or equal to 90% . In 2005, Kitamura of Kochi University in Japan used parallel binocular stereo vision technology to pick green peppers and made a prototype. The prototype consists of three parts: image processing, camera positioning, and cutting device. The image processing part includes two CCD cameras: an image capture card and an image processing application. When the images are acquired, the two cameras are placed in parallel. In the image recognition part, the prototype integrates illumination and stereo vision technology for image processing, and in the cutting part, it is controlled by a camera positioning system and a visual feedback system . In 2015, Ivorra et al. proposed the use of binocular stereo vision technology to assess the quality of mature grapes. The method obtains a three-dimensional model of clustered grapes by binocular stereo vision and then evaluates the quality of mature grapes using a SVM model based on the new three-dimensional descriptor. The algorithm has a good evaluation effect in the laboratory environment, but the evaluation effect in the actual environment is not known . Compared with foreign-related research, domestic application of binocular vision technology to fruit and vegetable picking robots started late, but with the efforts of some scholars, it has made some progress . In 2001, Zhang Ruihe et al. segmented the tomato image according to the two-dimensional histogram curve, then performed the area-based stereo matching, and finally calculated the spatial coordinate of the target. Through experiments, it can be concluded that when the distance between the target and the camera is 300~400 mm, the error between the theoretical value and the actual value is 3~4% . In 2010, Si Yongsheng and others used a normalized red-green difference to segment the kiwifruit to obtain the outline of the kiwifruit and then used the random ring method to extract the center and radius of the kiwifruit and used the limit constraint to perform horological matching based on area features on kiwifruit images. At the same time, kiwifruit with similar size is matched by order consistency constraint. According to the experiment of 130 images, the recognition success rate of the algorithm is 92%, which is in the range of 60~150 cm, and the error of measurement depth information is less than 2 cm . In 2015, Guo Aixia and others combined Harris Corner Detection with binocular stereo matching based on cosine distance to locate litchi picking points. The experimental results show that the algorithm can meet the complex operation requirements of litchi needs to be picked in series, and the matching success rate is 89.55% .
Through the above analysis, it can be seen that image recognition has achieved certain achievements in picking fruits and vegetables, but there are still few studies on kiwifruit, and kiwifruit is more special. Therefore, it is necessary to analyze the actual situation and design of a fruit recognition method suitable for kiwifruit to make up for the blank of this aspect. Based on the image recognition technology , this study analyzes the characteristics of kiwifruit and combines image recognition to realize the effective recognition and localization of kiwifruit, which can lay a foundation for subsequent kiwifruit picking.
2.1 Image processing
In the RGB color space, the kiwi leaves and ground weeds cannot be separated in the R, G, and B component images. At the same time, due to the scaffolding cultivation mode of kiwifruit trees, the orchard environment is heavily shadowed, so the grass, leaves, and trunks are mixed and cannot be separated in subsequent processing. In the HSV space, the boundary between the kiwifruit tree ridge and the grass on the ground and the leaves in the H component of the kiwi orchard image is clear, so it is easy to extract the tree row information and also suppress the influence of the shadow. The middle and upper branches of the S component are the same feature as the grassland, but the trunk and the ridge are both fuzzy, so it is not easy to extract the kiwifruit tree row. Meanwhile, the tree row information of the V component is closer to the ground than the ground. From the above analysis, it is known that the H channel grayscale image in the HSV space can effectively separate the ridges of the kiwifruit tree from the background, so this study converts RGB to HSV. The specific process is as follows:
2.2 Image enhancement
where G, F, and H are the Fourier transforms of the functions g(i, j), f(i, j), and h(i, j), respectively, and H(u, v) is the filter function, that is, the transfer function. The image filtering process can be divided into the following three steps: (1) The original kiwi orchard image f(i, j) is taken by Fourier transform to obtain F(i, j). (2) F(i, j) and H(u, v) are taken convolution to get G(u, v). (3) The inverse Fourier transform is performed on G(u, v), so that the enhanced image g(i, j) can be obtained.
2.4 Image segmentation
For an image with a distinct trough of the grayscale histogram, we first select an approximate threshold N (close to the value of the trough) to split the image into two parts, X1 and X2. Thereafter, the gray value average values ε1 and ε2 of the two regions are calculated, and the new threshold value is selected as N = (ε1 + ε2)/2 and the above process is repeated until the two gray value values are the same.
Among them, σi is the standard deviation of the gray value of the trunk region in the ith orchard image, and pi represents the gray value of the pixel selected in the region.
After the homomorphic filtering of the kiwifruit orchard image, the gray value of the kiwifruit trunk area is almost 0, and due to the change of light, the gray value of each trunk area in each image is distributed between 10.35. Therefore, combined with the actual situation, the average gray value of the kiwi trunk area of the eight sample images selected by this study is 3.81, and the threshold is T′ = 30. After the seed point is selected, the determination of the growth criteria is very important in the next.
The pixels of the peach orchard image are judged by similarity. The growth criteria have the following main steps: (1) The kiwi orchard image after preprocessing is scanned to find pixels that have not yet belong. (2) Focusing on the scanned home pixel, its domain pixels are grown and checked according to the four neighborhoods (up, down, left, and right) of the point. If the grayscale differences are less than a predetermined threshold, they are merged. (3) We start growing from the selected seed point until the entire image is processed and the seed point cannot be found or until the area cannot be further expanded. (4) The above steps are repeated until all the pixels are assigned and the growth is completed.
2.5 Fruit positioning
In the formula, i and j represent the horizontal and vertical coordinates of the fruit image pixel, respectively; n represents the total number of pixels of the fruit image; and Ω represents the set of pixels belonging to the same fruit image.
In the formula, x0 and y0 indicate that the fruit centroid coordinates xc and yc represent the image center coordinates.
In order to study the accuracy of the identification and localization method of kiwifruit in this study, it is now analyzed through experimental research. The operating environment of this test is as follows: hardware environment—operating system Windows7, processor AMD Athlon (tm) × 2DualCoreQL-64 2.10 GHz, memory 2GB; software environment—Matlab R2013a.
Performance comparison of kiwifruit recognition and localization algorithm
Algorithm of this study
Reaction speed (%)
Picking accuracy (%)
Missing picking rate (%)
Ripening rate of picked fruit (%)
Images collected in nature often cause noise due to equipment and other reasons, which may cause interference to subsequent image analysis and affect the accuracy of the analysis. Therefore, this paper uses the color model filtering method commonly used in the spatial domain to remove the noise of the acquired image, which lays a foundation for subsequent image processing. In addition, this article describes several common image color spaces and analyzes their relationship to each other.
Before the rapid dynamic recognition of overlapping kiwifruit, it needs to be segmented first, and the effect of segmentation directly affects the accuracy of recognition. After comparing several image segmentation methods, the paper finally adopts color difference segmentation algorithm of the improved color model. The algorithm performs gamma transformation on the R component in the image to stretch or shrink the R component in the image, thereby effectively alleviating the over-segmentation or under-segmentation and improving the image segmentation effect. The segmented image has noise and holes. Therefore, this paper combines mathematical morphology, hole filling, and threshold area retention method to further improve the denoising of the segmented image.
Considering that the kiwi picking robot is picking up during operation, this paper proposes an algorithm for rapid dynamic recognition of overlapping kiwifruit based on the correlation and difference of image sequences. First, a set of overlapping kiwi image sequences is acquired in a natural scene. After the segmentation is completed, a complete fruit contour can be obtained, and the position of the center of the circle is determined by finding the maximum value of the minimum distance from the point inside the contour to the edge of the contour. Then, the maximum distance from the center of the circle to the edge of the contour in different directions is found, and the minimum value is used as the radius. After that, the subsequent matching template is extracted according to the center of the circle and the radius information. Thereafter, the center position of each image in the continuously acquired image sequence is determined. At the same time, the two centers are respectively polynomial fitting, the motion path of the robot is fitted, the motion is predicted, and the position of the center of the circle in the next frame is estimated. Next, the processing area of the subsequent image is intercepted according to the center of the circle and the radius information. Finally, the matching identification of overlapping kiwifruit was carried out by using fast normalized cross-correlation matching.
This study verifies the effectiveness of the proposed algorithm through a series of experiments. First, the overlapping kiwifruit rapid dynamic recognition experiment was carried out. According to the experiment, it can be concluded that in this study, the trunk of the kiwifruit has been identified as the recognition object in the algorithm construction, so it can be eliminated with the background in the actual recognition. However, from the recognition result, some noise still exists, so it is necessary to perform noise removal processing on the image, thereby obtaining a better recognition effect. Through the noise elimination process, the fruit has basically been separated from the background of the picture, but there are still three pictures of two more overlapping kiwifruits in the picture, which need to be layered. The kiwifruit localization treatment was carried out in combination with the localization method of this study. The algorithm of this study separately locates the upper kiwifruit, and after system determination, the kiwifruit is located and the picking sequence is calibrated, which can effectively determine the fruit picking order. Finally, the performance comparison shows that the research method has certain validity for kiwifruit picking and can be applied to practice.
In the natural growth state, the overlapping of kiwifruit is very common. However, this phenomenon causes difficulties in the picking operation of the kiwi picking robot and affects the picking efficiency. Based on this, this paper proposes a method for rapid dynamic recognition and localization of overlapping kiwifruit. Firstly, the rapid dynamic recognition of overlapping kiwifruit is realized according to the correlation between image sequences. Secondly, the method of locating overlapping kiwifruit was studied. This paper analyzes how to overcome the influence of light intensity changes on the processing of kiwifruit orchard images, so that the kiwifruit trunk can be effectively segmented. At the same time, this paper chooses homomorphic filtering to achieve the purpose of image enhancement of kiwifruit orchard, highlighting the trunk characteristics of kiwifruit and reducing the influence of background noise on the identification of kiwifruit trunk. In , the image segmentation based on kiwifruit orchard and kiwi trunk is the important content of this paper. The study found that the method of this study can effectively separate the kiwifruit. Finally, the effectiveness of the research method is verified by experiments. According to the performance comparison, the research method has certain effectiveness on the picking of kiwifruit and can be applied to practice.
The authors thank the editor and anonymous reviewers for their helpful comments and valuable suggestions.
Availability of data and materials
Please contact author for data requests.
All authors took part in the discussion of the work described in this paper. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- C. Yongjie, Recognition and feature extraction of kiwifruit in natural environment based on machine vision. Trans. Chin. Soc. Agric. Mach. 44(5), 247–252 (2013)Google Scholar
- W. Zhan, D. He, S. Shi, Recognition of kiwifruit in field based on Adaboost algorithm. Trans. Chin. Soc. Agric. Eng. 29(23), 140–146 (2013)Google Scholar
- Q. Chi, Z. Wang, T. Yang, et al., Recognition of early hidden bruises on kiwifruits based on near-infrared hyperspectral imaging technology. Trans. Chin. Soc. Agric. Mach. 46(3), 235–241 (2015)Google Scholar
- L. Fu, Y. Feng, T. Elkamil, et al., Image recognition method of multi-cluster kiwifruit in field based on convolutional neural networks. Trans. Chin. Soc. Agric. Eng. 34(2), 205-211 (2018)Google Scholar
- L. Fu, W. Bin, C. Yongjie, et al., Kiwifruit recognition at nighttime using artificial lighting based on machine vision. Int. J. Agric. Biol. Eng. 2015(4), 52–59 (2015)Google Scholar
- G. Wenchuan, Early recognition of bruised kiwifruit based on near infrared diffuse reflectance spectroscopy. Trans. Chin. Soc. Agric. Mach. 44(2), 142–146 (2013)Google Scholar
- A. Moreno Álvarez, L. Sexto, L. Bardina, et al., Kiwifruit allergy in children: characterization of main allergens and patterns of recognition. Children 2(4), 424–438 (2015)View ArticleGoogle Scholar
- T.M. Le, M. Bublin, H. Breiteneder, et al., Kiwifruit allergy across Europe: clinical manifestation and IgE recognition patterns to kiwifruit allergens. J. Allergy Clin. Immunol. 131(1), 164–171 (2013)View ArticleGoogle Scholar
- R.M. Goodwin, N.M. Congdon, Recognition and attractiveness of staminate and pistillate kiwifruit flowers (Actinidia deliciosa var. deliciosa) by honey bees (Apis mellifera L.). N. Z. J. Crop Hortic. Sci. 46(1), 1–9 (2017)Google Scholar
- C. Nilsson, P. Brostedt, J. Hidman, et al., Recognition pattern of kiwi seed storage proteins in kiwifruit-allergic children. Pediatr. Allergy Immunol. 26(8), 817–820 (2015)View ArticleGoogle Scholar
- A.M. Twidle, F. Mas, A.R. Harper, et al., Kiwifruit flower odor perception and recognition by honey bees, Apis mellifera. J. Agric. Food Chem. 63(23), 5597 (2015)View ArticleGoogle Scholar
- D. Liu, W. Guo, D. Liu, et al., Identification of kiwifruits treated with exogenous plant growth regulator using near-infrared hyperspectral reflectance imaging. Food Anal. Methods 8(1), 164–172 (2015)MathSciNetView ArticleGoogle Scholar
- L. Qiang, S. Lili, C. Lili, A new approach for the detection of kiwifruit maturity by electrical impedance. App. Mech. Mater. 401-403, 1287–1294 (2013)View ArticleGoogle Scholar
- H. Zhu, B. Chu, Y. Fan, et al., Hyperspectral imaging for predicting the internal quality of kiwifruits based on variable selection algorithms and chemometric models. Sci. Rep. 7(1), 7845 (2017)View ArticleGoogle Scholar
- X. Dong, F. Wu, X.Y. Jing, Semi-supervised multiple kernel intact discriminant space learning for image recognition. Neural Comput. & Applic., 1–18 (2018)Google Scholar
- S. Baolin, Molecular detection of pv. on bacterial canker of kiwifruit. Acta Phytopathol. Sin. 43(5), 458–466 (2013)Google Scholar
- L. Fu, S. Sun, R. Li, et al., Classification of kiwifruit grades based on fruit shape using a single camera. Sensors 16(7), 1012 (2016)View ArticleGoogle Scholar