MNet: A Framework to Reduce Fruit Image Misclassification

India stood at the second position in the world in fruit producing after China. It is also ranked top in the list of countries who export fruits. In agriculture market as a whole, fruit markets has the highest economical weightage / share in the view of international or local market. Table 1 shows the statistics of the top fruits exported from India [1], along with the revenue generated from its sale. Thus while building our dataset high revenue generating fruits were considered. Farmers produce fruits, and then they either export the production to the international markets, or sell it to industries or to local market vendors. The fruits are processed in industry and different products like juice, pickles, jam, etc. are made which are either exported or sold at the market. Local market vendors sell the products to either small shopkeepers, supermarkets or directly to the customers. Thus, the three major categories of stakeholders for agriculture products are farmers, industries, and end-consumers or customers. Good fruit quality is the basic requirement of each stakeholder. The profit of business directly depends on the quality of fruit. Choosing good fruits in less time with high accuracy is the most vital requirement of the fruit business. The solution to this requirement must be user friendly and low cost which is affordable by all stakeholders of the agriculture products.

In India, fruit sorting is still being done manually, as shown in Figure 1. The manual forms of fruit classification/sorting/ grading can be very grueling, tiresome, back aching, and error prone, which directly affect the profit margin of stakeholders. Thus, by introducing technology to assist in fruit sorting as per the quality of grading, the manual stress required for the same task can be reduced and the efficiency will be increased. Every stakeholder at this end wants the good quality fruit and ignores the bad quality fruits. A good and bad quality fruit can be defined as: Good quality Fruit: a fruit having the required qualities or beneficial to stakeholders can be categorized as good quality fruits. Bad quality fruit: fruit of poor quality or low standard which is not desirable by stakeholders can be categorized as bad quality fruits. Consumers always desire the value of money while purchasing fruits from market.

The quality attributes of fresh fruits can be classified into three classes based on the occurrence of product characteristics as external, internal, and hidden respectively [2]. In the external class appearance (sight), feel (touch), and defects are considered as a quality attribute. In the internal class odor, taste, and texture are considered as a quality attribute. Many research works have been published on fruit classification based on internal and external quality attributes of fruits. The literature survey shows that systems based on internal classification are mostly sensor-based, complex, and costly because of extra hardware like NIR sensors.

Table 1. Top fruits exported from India

1.png

Figure 1. Manual sorting of pomegranate in the wholesale fruit market, Pune, Maharashtra, India

Most recently, Ünal [3] and Meshram et al. [4] show that the emphasis has been on the development of computer vision systems based on machine learning and deep learning algorithms to solve problems in the agriculture field. Mostly these systems consider the external quality attributes like size, shape, gloss, and color for fruit classification. These systems have promising results; they are less complex, user friendly, and fast. In this work, the external parameters of fruits are considered for classification.

Singh et al. [5] presented a multilayer convolution neural network (MCNN) to classify the mango tree leaves infected by the anthracnose fungal disease with creating an own dataset consisting of 1070 images of mango tree leaves. The obtained model achieved 97.13% accuracy. Kour and Arora [6] proposed a novel model called Fuzzy Rule-Based Approach for Disease Detection (FRADD) for apple fruit disease detection (called as apple scab) and classification. The Apple image dataset was created using a digital camera with different angles and distant values. The model achieved 91.66% accuracy. A new simple, effective, and lightweight model called DenseNet-16 was proposed to identify images of citrus diseases and pests in ref. [7]. The author created his own dataset of total images 12,562 (citrus pests = 9051 and citrus diseases = 3510) including seven different types of diseases and 17 species of citrus pests. The classification accuracy of the proposed model was 93.33%. Doh et al. [8] also worked on the automatic detection of citrus fruit diseases. Three machine learning algorithms K-means, SVM, and ANN were used in the proposed solution. The proposed solution uses color and texture features for classification. Hua et al., [9], presented a detailed review on latest advances in robots used for fruit harvesting. Use of robots for harvesting helps to increase the production and reduces the extra costs for apples, kiwifruit, sweet pepper, and tomatoes. Bresilla et al. [10] used a Convolution Neural Network (CNN) model with state-of-the-art, real-time object detection technique called You Only Look Ones (YOLO) for on-tree fruit detection.  The authors created his own dataset manually labeling 5000 images of pear and apple fruits. The model achieved more than 90% accuracy. Hossain et al. [11] proposed two models with different frameworks to classify the fruits into 15 and 10 classes, respectively. The first model was built with six layers while the second was fine-tuned visual geometry group-16 pretrained Deep Learning (DL) model. Two datasets were used to evaluate the model in this work. The first dataset used was public and consisted of 2633 images with 15 classes; whereas another was created by using internet images consisting of total 5946 images with 10 classes. Kirk et al. [12] proposed a Deep Neural Network (DNN) based strawberry fruit detection system which was tested on an existing dataset as well as on the real strawberry dataset. The model was evaluated with its F1 score, the harmonic mean of precision, and recall methods. Whereas Altaheri et al. [13] proposed a machine vision system to categorize the date fruit. Transfer learning from two famous CNN models, AlexNet and VGGNet were used to build three classification models to classify date fruit according to their maturity stage, type, and whether they are harvestable or not. Own dataset was created consisting of 8000 date images of five verities of dates in different maturity and prematurity stages.  To conduct the Precision Agriculture (PA) practice with the objective of increasing yield and crop marketability before harvest, Bauer et al. [14] developed a platform that chains up-to-date ML techniques, modern computer vision, and integrated software engineering practices to measure yield-related phenotypes from ultra-large aerial imagery named as AirSurf. This system was modified by combining computer vision algorithms and DL based classification model trained with 100,000 labeled lettuce signals. Zhang et al. [15] developed a harvesting robot for autonomous harvesting which consisted a low-priced gripper and a technique for detection of cutting point. The purpose of the study was to develop an autonomous harvester system which can harvest any crop with a peduncle rather than damaging its flesh. Onishi et al. [16] proposed a new system consisting of Single Shot MultiBox Detector (SSD) and a stereo camera for autonomous detection and harvesting of fruits. The experiment was conducted on an apple tree called “Fuji”. Zhang et al. [17] developed two improved model, namely, GoogLeNet and Cifar10, to identify 9 types of maize leaf diseases. A dataset of 500 images used in the experimentation was built from different sources like Plant, Village, and Google website which was divided into nine classes. Augmentation techniques like rotation, grayscale were used before training and testing the dataset. As a result, models GoogLeNet and Cifar10 achieved 98.9% and 98.8% identification accuracy, respectively. Liu et al. [18] proposed a novel pipeline consisting of segmentation, 3D localization, and frame-to--to-frame tracking for accurately counting the fruits from images. FCN (Fully Convolutional Network) was used to obtain the accurate fruit segmentation in the first stage of pipeline. Then the Hungarian technique applied to the Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) tracker to track the fruit through images. In the last stage, they have used SfM for localizing the fruit to correct the counting. This model was evaluated on the orange and apple fruits dataset. For multiple fruit classification, a Pure Convolutional Neural Network (PCNN) was proposed by Kausar et al. [19]. This model was built using seven convolution layers and Global Average pooling (GAP) layer to overcome the problem of over fitting. The performance of PCNN was tested on Fruit 360 dataset and the result shows that PCNN achieved 98.88% fruit classification accuracy. Femling et al. [20] proposed a system to atomize the fruits and vegetables detection process in the retail market. This system was built by using hardware Raspberry Pi and camera Module v2 and ML techniques. Two CNN architectures, Inception and MobileNet, were used to classify 10 different verities of fruits or vegetables.

Zhang et al. [21] studied how the dataset augmentation methods are related to the predication results of the classification results. The training dataset was built by using 200 colour images of five maturity levels, collected during different daylight conditions. Two augmentation operations are proposed to increase the dataset, which will help to raise the final predication accuracy, namely, Geometric Transformation and Random Noise. Authors developed their own CNN based model to classify tomatoes based on ripeness. The model architecture consists of five layers for feature extraction and one fully connected layer for classification.  Cheng et al. [22] developed a system for early yield predication with ANN. A new approach was proposed by considering the parameters of fruits and tree canopy while building the model, Two back propagation neural network models for yield predication were developed, one for the early period after June drop and another for the maturing period respectively. Puttemans, et al. [23] developed a robust pipeline by linking an object classification framework with application-specific pre-filtering, scene and proficient cluster segmentation for strawberry and apple fruits detection. The correct output of the trained object detector is always depending on the input image annotations. Annotation biasing is the major problem which can affect the final detection accuracy. Stein et al. [24], developed an efficient yield estimation model using multisensory framework and the DL algorithm. The proposed system was able to detect, track, localize, and map every fruit in a mango farm. The latest DL algorithm called faster R-CNN was used to detect the fruits from images. To solve the problem of occlusion, a multiple viewpoint approach was used which avoids the need for labor intensive field calibration. 

Yamamoto et al. [25] developed an on-tree tomato fruit classifier using ML algorithms to classify it into mature, immature, and young tomatoes. Classification decision was based on the size, texture, shape, and colour parameters extracted from the images.  Font et al. [26] developed automated fruit harvester which consists of a low-priced camera and a gripper tool consisting of a robotic arm. In (Suchet Bargoti and James Underwood, [27]), a new object detection framework, Faster R-CNN was presented in the paper. The model was tested for detection of mangoes, almonds, and apple fruits in orchards. A tiling approach on images was introduced in the study, which improved the detection accuracy of Faster R-CNN with F1-score greater than 0:9 for mango and apples.

The DL based classifier to classify Cavendish banana grade was explored in Ucat et al. [28]. The model was developed using Python, OpenCV, and Tensorflow. The model achieved more than 90% classification accuracy. Ireri et al. [29] proposed a machine vision system for postharvest tomato grading. The system works on RGB images given as input to the system. Own datasets were created by manually labeling the tomato images into four categories according to their defect, healthy, and ripeness parameters. Four different models were built to classify the image into one of the categories according to the matching features. Total 15 features were considered while taking the decision. The result shows that RBF-SVM performed well compared to others for category 1 i.e. healthy or defected with 0.9709 detection accuracy. Piedada et al. [30] developed a system for banana (Musa acuminata AA Group 'Lakatan') classification using ML techniques based on tier-based. A noninvasive tier-based technique was used in this study. ANN, SVM, and RF classifiers were used to classify bananas into extra class, class I, class II, and rejected classes. The result showed that the random forest algorithm outperformed compared to others with 94.2% accuracy.  An automated real–time grading system for quality inspection for apple fruit was developed by Sofu et al. [31]. The developed system comprises a roller, transporter and class conveyor joined with an enclosed cabin with camera, load cell, and control panel units. The system not only classifies the apples on the basis of color, size, and weight parameters, but also identifies defective apples. A grading and sorting system based on machine vision for date fruit was developed by Yousef Al Ohali [32]. The system was able to categorize the dates into three classes (grade 1, 2, or 3) using the given RGB image as an input. A back-propagation algorithm was tested in the study which showed 80% accuracy. Fruits and vegetable quality depends on their parameters like shape, size, texture, color, and defects. Different methods have to apply to classify the fruits and vegetables according to their quality parameters like data collection, preprocessing of data, image segmentation, feature extraction, and finally classification.

The classification methods studied as above have one or more limitations such as:

1. The classification doesn’t take into account the needs of the real stakeholders and thus the proposed models are not helpful to farmers, industries, and customers.
2. Few works are either trained on public dataset or images from internet instead of creating own datasets, as in Doh et al. [8], Hossain et al. [11], Onishi et al. [16], Zhang et al. [17], Kausar et al. [19], and Bargoti et al. [27].
3. The available public or private datasets consists of general classes of fruits and do not mention the quality of fruits.
4. Few classification systems are costly as they require extra hardware, as in Femling et al. [20].

To overcome these limitations, this paper presents a framework which makes the model robust, efficient, and accurate for fruit classification. The major contributions of this paper are as follows:

1. Published own dataset of top Indian fruits consists of more than 12000 colour images taken in different light conditions with different angles with quality labels [33].
2. Presented/Studied misclassification problems in prebuilt models.
3. Proposed novel framework to build a model called as MNet is proposed to solve the misclassification problems.
4. A popular pretrained model, InceptionV3 is used to test the framework on random images from Google and the results show that we are able to increase the accuracy and mitigate the problem of misclassification.

The rest of the paper is organized as follows: Section II presents the misclassification problem in popular published models. Section III describes in detail the dataset used for the experimentation. Section IV describes the proposed framework in detail. Section V reports the experimental results. Then discussion and conclusion with a summary and of future work given in section VI and Section VII.

The success of a research experiments depends on how large and clean dataset you have. It plays an essential role in success of the research project. It is very difficult to get ready-made dataset which you can use as it is in your project. Therefore, our major goal was to build a dataset that contained many fruit images, where each fruit is represented with many images as possible. Thus to build our own dataset we considered the top fruits exported from India [1] based on revenue generated, as shown in Table 1. Datasets can be used to solve two major problems, either used for object classification or for object category recognition and detection. For this work, we have created a dataset useful for object classification. The dataset was created in Vishwakarma University campus, Pune, India, in July and August 2019. The images were captured inside and outside the lab, at the very high resolution of either 2448×3264×3 or 3024×3024×3 pixels by using 13+ megapixel mobile rear cameras.

For experimental purpose, the images were down sampled to 256×256×3 resolution. The dataset comprises of six different categories with two subcategories of each and number of images for each class. Figure 2 shows the sample images from the dataset of each class. Images were collected at various times on different days for the same category with natural background. These features improve the dataset variability and represent a more realistic scenario. The images had large variations in quality, lighting, and background. Illumination is one of those variations in imagery, illumination can make two images of the same fruit less similar than two images of different kinds of fruits. The fruit dataset was collected under relatively unconstrained conditions. Images were also captured with different light conditions in the laboratory or outside in the sunlight with different background scenarios. Basic conditions which are considered during dataset collection are:

1. Fruits in different positions.
2. Popular categories of fruits in India for export and consumption.
3. Used high resolution mobile cameras to capture to images.
4. Different lighting conditions with different background.
5. Photos were taken inside and outside the lab.

Dataset of 12000 images [33] was created and used in the experimentation. ImageDataGenerator API in Keras is used for generating augmented images. This package overcomes the problem of small dataset.

The ImageDataGenerator package also provides functions for image pre-processing. For experimental purpose, the images were down sampled to 256×256×3 resolution. The dataset comprises of six different categories with two subcategories of each and number of images for each class. Figure 2 shows the sample images from the dataset of each class. Images were collected at various times on different days for the same category with natural background. These features improve the dataset variability and represent a more realistic scenario. The images had large variations in quality, lighting, and background. Illumination is one of those variations in imagery, illumination can make two images of the same fruit less similar than two images of different kinds of fruits. The fruit dataset was collected under relatively unconstrained conditions. Images were also captured with different light conditions in the laboratory or outside in the sunlight with different background scenarios.

Basic conditions which are considered during dataset collection are:

1. Fruits in different positions
2. Popular categories of fruits in India for export and consumption.
3. Used high resolution mobile cameras to capture to images.
4. Different lighting conditions with different background.
5. Photos were taken inside and outside the lab.

Dataset of 12000 images [33] was created and used in the experimentation. ImageDataGenerator API in Keras is used for generating augmented images. This package overcomes the problem of small dataset. The ImageDataGenerator package also provides functions for image pre-processing.

2.png

Figure 2. Examples of images from each class with resolution

The experiments are carried out to evaluate the performance of the model based on our proposed framework. The results of MFC_InceptionV3, which are based on our framework are compared with the FC_InceptionV3 which is built on the standard transfer learning technique. We first prepared a dataset by equally dividing 1000 images in each class. Total 12000 images divided into 12 classes are considered as shown in Figure 5. This assured balanced dataset for the experiment. The Google Colaboratory or "Colab" is used to write and run the experiments. We used GPU support provided by Google Cloud server. We used Keras API written in Python provided for image classification, which runs on top of the TensorFlow machine learning platform. Figures 3 and 4 explains the detail framework designed for fruit classification. Each architecture consists of three parts. The first part is to input images having shape (299, 299, 3). In the second part, we used transfer learning by using pretrained model weights by freezing half layers and retrained the model on our dataset. Here Why do we freeze the layers and then unfreeze again to train the model? Transfer learning is used when a model wants to get the knowledge of the pretrained model and use this knowledge to identify the required objects.

5.png

Figure 5. Total 12000 images were equally divided into 12 classes

This head-start of imparting knowledge into the model can be very helpful for the model. It is usually done for models where the dataset is small and the computational power is less, but it is a perfect way to train a model in an efficient manner. In the third part Global Average Pooling (GAP) is used and some dense layers are applied on the top layers along with a dropout of 25% to avoid overfitting. Dense layers are applied with relu and softmax as the activation function to classify the objects.

Global average pooling (GAP) layers are used instead of fully connected layers to avoid the overfitting. GAP layer reduced the parameters in the model which makes it more robust [37].

Out of several activation functions like linear function, Sigmoid Function (σ), Tanh Function; ReLU [38] is one of the most popular functions in many classification tasks. The function is denoted as Eq. (1).

$F(x)=\max (0, x)$     (1)

where, x represents the input to a neuron. The equation can also be represented as Eq. (2).

$f(x)=\sum \begin{array}{l}0, \text { for } x<0 \\ x, \text { for } x=>0\end{array}$     (2)

We exploited ReLU in our experiment; to activate it needs only a threshold value. As it doesn’t need extra calculations, it is faster and makes the model lighter. Another activation function is called softmax [39], which is very popular and used in all most all multiclass neural network models in the last layer. The function is denoted as Eq. (3).

$p_{j}=\frac{\exp \left(x_{i}\right)}{\sum_{i=1}^{k} \exp \left(x_{i}\right)}$      (3)

where, Pj represents the probability of each class, in our case, it is six. This returns the probabilities of each class and the class having the highest probability is our target or output class.

This section discusses the limitations of the presented work. In this work, an attempt is made to meet the current needs of the users, but still, we acknowledge a few limitations. We created our own dataset of 12000 images considering a top Indian fruit depending on consumption or exported. Each fruit can have subcategories which are not considered in the study. For example, “Pomegranate” is a general category of fruit, but in India four verities of pomegranate are popular, namely, Arakta, Bhagwa, Ganesh, and Ruby [40]. A standard method called data augmentation is used to overcome the small dataset problem.

Four popular models Inception, Resnet152V2, Xception, and MobileNet are evaluated, but the list of pretrained models is large. The output of the model showed the class of fruit with quality, for example “Apple Bad” or “Apple Good”, but it doesn’t do the fruit detection of the bunch of fruits. It also doesn’t show labels on the output. For this we can use the latest object detection algorithm called YOLO (you only look ones). The major limitation of this work is the solution which is computationally intensive, i.e., we need to create a separate model for each parameter. However, the advantages of this approach promise more accuracy, this reduces the misclassification problem, the models become modular. This approach is good when your dataset is small and has similar feature images.