This paper attempts to design an automatic grading system that can accurately and efficiently determine the size and surface quality of apples. For this purpose, apple images were processed by the LabVIEW toolkit (including IMAQ Vision), and the particle area analysis was performed to measure the size and evaluate the surface quality of apples, creating an apple size and surface quality detection and grading system. The proposed system can detect the size and surface defects of each apple, grade the surface quality, and transfer the apple to the storage point for apples in that grade. With clear principles, simple processes and good portability, the system can be applied independently or together with other detection methods to evaluate the internal quality of apples, such as sugar content, maturity and pesticide residue. The research findings make it possible to make a comprehensive evaluation of apple quality.
LabVIEW, particle area, surface quality, grading, manipulator and motion control
The grading of appearance quality is an operational step before fruit marketing, which has a significant impact on the sales price and planting prospect of fruit. About 32 million tons of apples are produced worldwide each year, but they vary in quality and must be graded according to the level of storage and sale. While traditional fruit classification relies on manual operation and judgment, with low technical requirements and is more suitable in the case of abundant human resources, and a possible alternative should be an automatic inspection and grading system of the apples, which exhibits high speed and efficiency.
Many studies have developed grading machines based on normal image to detect the physical appearance of apples [1-4], or applied methods of laser image, hyperspectral image, near-infrared light, Raman spectrum, and weak light to detect the internal quality of the apples [5-11]. In this paper, a mass detection method based on particle area analysis is proposed, the appearance of apples can be detected to rapidly determine the surface characteristics, like color, texture, and surface defects, since apples from the same region, orchard, and of the same variety have similar internal quality, so as to allow the grading of apples after picking and before storage.
The purpose of this study is to design an accurate and efficient grading system to detect the size and surface quality of apples and realize automatic grading. By means of image processing technology this quality inspection method based on particle area analysis, can determine the size of apples and their surface quality effectively, which allows direct grading. In this paper, the authors introduce the design idea and basic principles of the system, corresponding algorithm, and flow chart of grading manipulator control respectively in detail.
The following procedures are followed for surface quality inspection of the apples. Firstly, place the picked apples on the inspection platform, and use an industrial HD camera MV-200UC to capture the images of these apples, which will be analyzed, so as to judge the size and surface quality of the apples. Then, use the transfer manipulator to move the apples to separate buckets based on their quality [12-18].
The overall structure of the apple surface quality inspection system is shown in Figure 1, which consists of a digital camera MV-200UC, a USB image capture box MV-U2000, and a fixed focal length lens M1214-MP or a continuous zoom lens VS-1024. After installation of the lens, the camera is fixed on the test platform, so that its focal is fixed as well. The size of apples and distance of the apples to the lens vary, which can influence the image definition. Therefore, when there is no large difference in the outer dimensions of apples, the fixed focal length lens M1214-MP is selected, but if there is a large difference between the outer diameters of apples, the continuous zoom lens VS-1024 should be adopted. When taking pictures of apples, the authors use the circular LED vision VS-RL100 to illuminate the apples; and perform fine tuning of the lifting table of the test platform according to the size of apples and focal length, and the distance of fine tuning is displayed through the horizontal scale bar, which is convenient for the authors to read.
The manipulator that is used to execute apple grading is a 5DOF articulated robot with rotary joints. The waist, shoulder, and elbow joints decide the position of its hands, and the wrist pitch joint and wrist roll joint decide the pose of its hands. The structural schematic diagram is shown in Figure 2 [19-23]
The authors use servo stepper motors and Universal Motion Interface cards (UMI-7764 of National Instruments) of the same type for all joints. A single UMI-7764 motion control card can control four motors at the same time; therefore, the authors install two UMO-7764 motion control cards for the system in total. The system structure driven by the manipulator motors is depicted in Figure 3.
Figure 1. Apple surface quality inspection system
Figure 2. Schematic diagram of the joints of the manipulator
Figure 3. Driven of structure manipulator motor
The image processing function of the system covers image acquisition, display, and quality inspection of the apple surface. The LabVIEW and its image processing toolkit including IMAQ Vision are adopted to process images; and the processing procedures include both manual and automatic tests, with the former one being mainly adopted in the test and debugging of algorithm parameters. While the combined algorithm parameters can be used to obtain automatic test procedures and realize automatic inspection of apple surface quality.
Figure 4. Small apple with better surface quality
Figure 5. Inferior apple with worse surface quality
Figure 6. Medium-scale apple with better surface quality
Figure 7. Inspection procedures of images
The picked apples are placed on the inspection platform for image acquisition, and the surface of apples is illuminated with a light source to eliminate the influence of reflection on inspection, and to enhance the edge of images. The captured images are provided in Figure 4, Figure 5 and Figure 6 respectively.
The inspection procedures of images are exhibited in Figure 7. After image acquisition and display, the captured images are subject to pre-treatment and segmentation. Next, the surface quality inspection is performed. First of all, an assessment is made to detect if there is any defect on the surface; those with defects are classified as inferior apples; and for those with no defect, the particle size comparison is carried out using a preset particle size to determine the size of the apples. The apples are classified as big, medium, and small apples depending on the correlation of their particle size values to that of the preset one. If there is no artificial stop, the procedure would return to target recognition & detection and surface quality inspection, and continue inspection. If there is any artificial stop, the procedure would return to the starting point and then end.
The 32 bit RGB color image is converted into an 8 bit binary image for processing of captured apple images. Upon completion of the processing, a closed figure of gray level is 1 forms on the surface of apples with better surface quality, and the edge of the images is smooth and clear as shown in Figure 8. Other particles appear on the surface of apples with poor surface quality, such as, defects left by worms, insects, or other physical damages. As shown in Figure 9, particles in the small rectangular box are defects left by worms, collision, or puncture; while particles in the larger rectangular box is the concave part of the stem, which are not considered as defects.
Figure 8. Binary image of apples with better surface quality
Figure 9. Binary image of apples with poorer surface quality
Figure 10. Inspection procedures of apple surface quality
Therefore, the surface inspection mainly includes the inspection of apple size and surface defects that is performed with the assistance of the IMAQ Particle Analysis Report VI (particle analysis report). The Connectivity 4/8 in the VI refers to the connecting method between pixels (4-connecting or 8-connecting, and it is 8-connecting by default); the output includes the number of particles in pixels and as real length (in mm), and the Calibration Valid indicates whether the particles are calibrated effectively. The particle report includes the area, number of holes, bounding of the rectangle, center of mass, orientation, and dimension; however, only the area and number of holes are employed in the grading process currently.
Figure 11. Inspection results of medium apples with surface quality
Figure 12. Inspection results of small apples with better surface quality
Figure 13. Inspection results of big apples with surface defects
The particle area is taken as the measurement parameter, which can be extracted from the particle report, and the size of apples can be determined by comparing the maximum particle area with the pre-set one, allowing the classification by size. When the maximum particle area of an apple is larger than the pre-set area of big apples, the apple is classified as a big apple, and the manipulator for big apple classification starts. When the area of an apple is smaller than or equal to the pre-set area of small apples, the apple is classified as a small apple, and the manipulator for small apple classification starts. When the area is between the two preset areas, the apple is classified as a medium apple, and the manipulator is programmed to perform classification of medium apples. Similarly, the surface quality of defective apples can be determined by checking whether there is any defect on the surface, and by comparing defective particle area with the default area. If the defective particle area in the image of an apple is larger than the present value, it can be judged that the surface of the apple is defective, and the manipulator for substandard fruit classification starts. The inspection procedures of apple surface quality are shown in Figure 10.
The above foregoing inspection method of apple surface quality is used repeated with medium-scale apples and, small-scale apples with better surface quality, and apples with surface defects on the surface to verify the inspection procedures; and the results are as shown in Figures 11, Figure 12, and Figure 13, respectively.
The detectable quantities are necessary, including position feedback and speed feedback, as well as the limit switch and I/O passage for the safe operation of the manipulator, so as to realize accurate positioning, multi-axial synchronization, and the ability to move at a designated speed, acceleration and track. The overall scheme of the manipulator movement control feedback inspection is shown in Figure 14.
According to the data presented in Figure 2, Figure 3, and Figure 14 respectively, the structure of grading manipulator, manipulator motor driving structure, and manipulator motion feedback inspection principles can be applied to the mechanical system, control system and inspection system of grading manipulator of apples. Motion control of the manipulator is realized through combination of motors in all joints, using the Robotics toolkit in the LabVIEW software, Vision and Motion toolkit. The robot motion control card is connected with the inspection sensor to form a robot motion control system. The program simulates the procedures and motion control procedures of the manipulator, and then applies the simulated results of motor speed, acceleration and speed adjustment to the motion control procedures of the manipulator, so as to quickly develop the grading control procedures of the grading manipulator (which are not described in detail herein due to length limitation). The motion control systems of motors are similar. And the basic principles of manipulator motion control are shown in Figure 15.
Figure 14. Feedback inspection principles of manipulator motion
Figure 15. Schematic of the motion control principles of manipulator
The authors apply the LabVIEW and IMAQ Vision tools for the surface quality inspection of apples; propose an inspection method of apple size and surface quality based on particle area analysis, and also describe the motion control of the grading manipulator as well. The principles and procedures of the processing method are simple, and the algorithm has good portability. Thus, this surface quality inspection method of the apples can be used alone or together with other inspection methods that are used to assess the internal quality of the apples, such as, sugar content, maturity, and pesticide residue, so as to realize more comprehensive evaluation of apple quality.
This research has received support from the Fundamental Research Funds for the Central Universities (No. 2452016077), Shaanxi Key R&D Program (2019NY-171, 2018TSCXL-NY-05-04). And Students’ Innovative Research Plan of Northwest A&F University (No. 1201710712048). The authors are also gratefully to the reviewers for their helpful comments and recommendations, which make the presentation better.
 Sofu, M.M., Er, O., Kayacan, M.C. (2016). Design of an automatic apple sorting system using machine vision. Computers & Electronics in Agriculture, 127(C): 395-405. https://doi.org/10.1016/j.compag.2016.06.030
 Shilts, P., Motabar, P., Lefcourt, A.M., Kim, M.S., Tasch, U. (2014). Homogenization optics to improve detectability of fluorescence response to a single laser pulse. Detection of Feces on Apples, 57(2): 615-622. https://doi.org/1 0.13031/trans.57.10443
 Trincă, L.C., Căpraru, A.M., Arotăriţei, D., Volf, I., Chiruţă, C. (2014). Monitoring methods and predictive models for water status in jonathan apples. Food Chemistry, 144(2): 80-86. https://doi.org/10.1016/j.foodchem.2013.05.131
 Vanoli, M., Rizzolo, A., Grassi, M., Spinelli, L., Verlinden, B.E., Torricelli, A. (2014). Studies on classification models to discriminate ‘braeburn’ apples affected by internal browning using the optical properties measured by time-resolved reflectance spectroscopy. Postharvest Biology & Technology, 91: 112-121. https://doi.org/10.1016/j.postharvbio.2014.01.002
 Baranowski, P., Mazurek, W., Wozniak, J., Majewska, U. (2012). Detection of early bruises in apples using hyperspectral data and thermal imaging. Journal of Food Engineering, 110(3): 345-355. https://doi.org/10.1016/j.jfoodeng.2011.12.038
 Chiu, Y.C., Chou, X.L., Grift, T.E., Chen, M.T. (2015). Automated detection of mechanically induced bruise areas in golden delicious apples using fluorescence imagery. Transactions of the ASABE, 58(2): 215-225. https://doi.org/10.13031/trans.58.10578
 Zhang, B., Fan, S., Li, J., Huang, W., Zhao, C., Qian, M. (2015). Detection of early rottenness on apples by using hyperspectral imaging combined with spectral analysis and image processing. Food Analytical Methods, 8(8): 2075-2086. https://doi.org/10.1007/s12161-015-0097-7
 Niu, L., Zhou, W., Wang, D., He, D., Zhang, H., Song, H. (2016). Extracting the symmetry axes of partially occluded single apples in natural scene using convex hull theory and shape context algorithm. Multimedia Tools & Applications, 76(12): 14075-14089. https://doi.org/10.1007/s11042-016-3781-8
 Zhang, B., Huang, W., Liang, G., Li, J., Zhao, C., Liu, C. (2015). Computer vision detection of defective apples using automatic lightness correction and weighted RVM classifier. Journal of Food Engineering, 146: 143-151. https://doi.org/10.1016/j.jfoodeng.2014.08.024
 Yuan, L.M., Cai, J.R., Sun, L., Han, E., Ernest, T. (2016). Nondestructive measurement of soluble solids content in apples by a portable fruit analyzer. Food Analytical Methods, 9(3): 785-794. https://doi.org/10.1007/s12161-015-0251-2
 Zhang, B., Huang, W., Wang, C., Gong, L., Zhao, C., Liu, C. (2015). Computer vision recognition of stem and calyx in apples using near-infrared linear-array structured light and 3D reconstruction. Biosystems Engineering, 139: 25-34. https://doi.org/10.1016/j.biosystemseng.2015.07.011
 Li, Q., Wang, M., Gu, W. (2002). Computer vision based system for apple surface defect detection. Computers & Electronics in Agriculture, 36(2): 215-223. https://doi.org/10.1016/s0168-1699(02)00093-5
 Zhang, B.H., Li, J.B., Zheng, L., Huang, W.Q., Fan, S.X., Zhao, C.J. (2015). Development of a hyperspectral imaging system for the early detection of apple rottenness caused by penicillium. Journal of Food Process Engineering, 38(5): 499-509. https://doi.org/10.1111/jfpe.12180
 Golpira, H., Golpîra, H. (2016). Redesign and evaluation of an automatic fruit sorter. International Journal of Advanced Manufacturing Technology, 89: 1791-1798. https://doi.org/10.1007/s00170-016-9212-1
 Ji, W., Qian, Z., Xu, B., Tao, Y., Zhao, D., Ding, S. (2016). Apple tree branch segmentation from images with small gray-level difference for agricultural harvesting robot., Optik. International Journal for Light and Electron Optics, 127(23): 11173-11182. https://doi.org/10.1016/j.ijleo.2016.09.044
 Masoumi, A.A., Kalhor, M., Shafaei, S.M. (2015). Design, construction and evaluation of an automatic apple grading system. Agricultural Engineering International: The CIGR E-Journal, 17(1): 247-254.
 Xiang, R., Jiang, H., Ying, Y. (2014). Recognition of clustered tomatoes based on binocular stereo vision. Computers and Electronics in Agriculture, 106: 75-90. https://doi.org/10.1016/j.compag.2014.05.006
 Li, X., Zhu, W. (2011). Apple grading method based on features fusion of size, shape and color. Procedia Engineering, 15: 2885-2891. https://doi.org/10.1016/j.proeng.2011.08.543
 Ji, W., Zhao, D., Cheng, F., Xu, B., Zhang, Y., Wang, J. (2012). Automatic recognition vision system guided for apple harvesting robot. Computers & Electrical Engineering, 38(5): 1186-1195. https://doi.org/10.1016/j.compeleceng.2011.11.005
 Bac, C.W., Henten, E.J., Hemming, J., Edan, Y. (2015). Harvesting robots for high-value crops: State-of-the-art review and challenges ahead. Journal of Field Robotics, 31(6): 888-911. https://doi.org/10.1002/rob.21525
 Zhao, D., Lv, J., Ji, W., Zhang, Y., Chen, Y. (2011). Design and control of an apple harvesting robot. Biosystems Engineering, 110(2): 112-122. https://doi.org/10.1016/j.biosystemseng.2011.07.005
 Bulanon, D.M., Kataoka, T. (2010). Fruit detection system and an end effector for robotic harvesting of Fuji apples. Agricultural Engineering International: The CIGR E-Journal, 12(1): 203-210.
 Kreutz-Delgado, K., Long, M., Seraji, H. (1992). Kinematic analysis of 7-dof manipulators. International Journal of Robotics Research, 11(5): 469-481. https://doi.org/10.1109/ROBOT.1990.126090