Fatih Özyurt* | Eser Sert | Derya Avci
© 2022 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
The need for automatic disease detection applications that can help farmers in the detection of agricultural product diseases is increasing day by day. Convolutional Neural Network (CNN) is a very popular field in image processing, recognition, and classification. It is seen that CNN architectures are used in the determination of agricultural products. In this study, 3 different ResNet architectures of the features automatically are used in the detection of tomato diseases. The most efficient features obtained from these architectures have been obtained by the NCA algorithm again. The features obtained have been trained with the Cubic SVM machine learning algorithm. Tomato leaves belonging to a total of 10 classes have been trained at 80% and a test performance rate of 98.2% has been achieved.
residual network, NCA, tomato leaf disease, deep learning
Tomato is one of the most consumed food items all over the world because it is the main ingredient of many food types. Tomato productivity is very important for both tomato farmers, marketers, and consumers. Tomato plants can be affected very rapidly by different diseases at every stage of growth. Especially climatic conditions and environmental factors are the most important factors triggering these diseases. This can lead to serious reductions in tomato production yield [1]. To minimize these disadvantages, machine learning-based approaches that detect these diseases at an early stage are important. In this study, a classification approach that can detect 9 basic tomato disease types is proposed. These are tomato yellow leaf curl virus, tomato mosaic virus, target spot, spider mites, septoria spot, leaf mold, late blight, early blight, bacterial spot. In this study, a classification approach based on a convolutional neural network (CNN) and Support Vector Machine (SVM) has been proposed. The working structure of the systems that automatically perform the detection of the diseased areas in the leaves may vary according to the machine learning basis used.
Various studies in the literature can classify the disease by supervised and unsupervised structures. Both supervised-based and unsupervised-based plant disease classification system approaches enable the identification of plant diseases in large fields and take measures for the farmer. It can also make a great contribution to minimizing the symptoms that occur.
As can be seen from Table 1, different studies have been conducted in the literature to identify and classify diseases in tomato leaves [2-14]. In the detection of diseased areas in tomato leaves, The Grey level co-occurrence matrix (GLCM) and ROC curve analysis method developed by Lu et al. [2] has 100% classification success. However, only Yellow Leaf Curl type tomato leaf disease was detected in this study. 9 types of tomato leaf diseases were classified as CNN-based. In this study, 99.18% success was achieved by Brahimi et al. [3].
In the study conducted by Karthik et al., 3 types of tomato leaf diseases such as Early Blight, Late Blight and Leaf Mold were detected by CNN approach [4]. In this approach, two different deep architectures were used to determine the type of infection in tomato leaves. In the study, 98% classification success was achieved. In the study conducted by Zhang et al. (2018), 8 types of tomato leaf diseases such as Early Blight, Yellow Leaf Curl Disease, Corynespora Leaf Spot Disease, Late Blight, Septoria Leaf Spot, Two-Spotted Spider Mite were identified using CNN architecture. AlexNet, GoogLeNet and ResNet were used in the study [5]. They found the highest accuracy of 97.28% for identifying tomato leaf disease by using the ResNet model. In the study proposed by Suryawati et al., Alexnet, GoogleNet and VGGnet were used, respectively. In this study, the highest result was obtained with VGGnet and the success rate was 95.24% [6]. Tm and colleques, proposed a CNN-based approach for the classification of pest in tomato plants. The proposed system has achieved an average accuracy of 94-95% [7]. It can be seen from Table 1 that many studies have been carried out using deep learning methods [8-11].
Additionally, different methods other than deep learning methods were used. As can be seen from Table 1, the Fuzzy Inference System (FIS), the Adaptive neuro-fuzzy inference system, the multi-layer feed forward back propagation neural network (MLBPNN), Support Vector Machine (SVM), Decision Tree based on the tomato leaves were classified as diseased regions [12-14].
Deep Learning is a tremendous technology and it is a method that is needed in almost every field. Many studies such as leaf recognition, face recognition, voice recognition, biomedical image recognition, satellite data recognition, etc. can be done with Deep learning [16-21].
Table 1. Various approaches for classification of diseases in tomato leaves
|
Publication and Year |
Approach |
|
Lu et al., 2018 [2] |
Grey level co-occurrence matrix (GLCM) and ROC curve analysis |
|
Brahimi et al., 2017 [3]; Karthik et al. 2019 [4]; Zhang et al., 2018 [5]; Suryawati et al., 2018 [6]; TM et al. , 2018 [7]; Sardogan et al., 2018 [8]; Jia et al., 2017 [9]; Fuentes et al. [10]; Durmus et al., 2017 [11] |
Deep Learning (CNN) |
|
Sabrol and Kumar, 2016 [12] |
Fuzzy Inference System (FIS), Adaptive neuro-fuzzy inference system, multi-layer feed forward back propagation neural network (MLBPNN) |
|
Raza et al. 2015 [13] |
Support Vector Machine (SVM) |
|
Sabrol and Kumar, 2016 [14] |
Decision Tree |
Although various approaches have been presented in the literature so far for the detection of diseases in tomato leaves, there are still some difficulties. This article has been proposed for the following research gaps and motivations.
1. In order to distinguish diseases in plant leaves in traditional image processing techniques, the features must be extracted by handcrafted methods. Differences in diseases of plant leaves cause differences in disease patterns. In this case, the feature extraction method suitable for a disease type may not be suitable for a different disease pattern. Therefore, feature extraction methods are required to handle the features of patterns at pixel level.
2. The performance of the models based on machine learning can vary depending on the number of data and the number of features obtained from these data. The excess number of samples increases the runtime in case of manual feature extraction. Therefore, selecting or extracting of features must be performed automatically.
3. Deep learning models such as ResNet-18, ResNet-50, ResNet-101 are general and proven architectures. In the classical usage of these architectures, problems such as high capacity hardware sufficiency problems, long working times, and low data problems are encountered.
4. In addition, most of the studies using CNNs use classical fully connected pretrained CNNs. For this reason, the contribution rate of pretrained CNN architecture used in the proposed methods to the literature cannot be measured exactly.
Due to the gaps in the literature described above, the following main contributions are presented in the proposed study.
1. ResNet-18, ResNet-50, ResNet-101 CNN's architectures are used in this study. The main purpose of using these three architectures is to take advantage of the advantages of ResNet architectures in feature extraction.
2. The features obtained from ResNet-18, ResNet-50 and ResNet-101 architectures have been reduced by NCA algorithm and 1000 features with the highest representation have been selected.
3. In the literature, a better method of detecting diseases in tomato leaves with such high success and low working time has not been proposed.
4. In the proposed architecture, a total of 7500 images belonging to 10 classes have been used. 80% of these images have been used for education and 20% of images have been used for testing. By applying augmentation only to images belonging to the mosaic virus class, the number of images has been increased from 373 to 750 images. 750 images of the other 9 classes have been used from the database. By augmentation, the image numbers have been equalized, thus preventing the wrong classification accuracy from being increased.
This paper is organized as follows. Following the introductory part, the theoretical background is presented in Section 2. The proposed approach and experimental results are given in Section 3 and Section 4, respectively. In section 5, the conclusion is presented.
2.1 Database
In this study, pictures of tomatoes belonging to 10 classes in the Plant Village dataset have been used [15]. These classes and the number of images of these classes are given in Table 2. The total number of images in the data set is 16011. As shown in Table 2, the class with the lowest number is the mosaic virus class with 373 images. Only the data for this class have been increased from 373 images to 750 images by 180 degrees rotating and flipping augmentation. Flipping means rotating an image on a horizontal or vertical axis. In horizontal flip, the flipping will be on the vertical axis, In Vertical flip, the flipping will be on the horizontal axis. Rotation means rotates the image by a specified degree. Figure 1 shows the image Augmentation methods of the Mosaic virus class. To maintain the balance between data, only 750 images of each class have been used.
750 images of each class have been used to evaluate the proposed model, 7500 images belonging to 10 classes have been used in total. 80% of these data have been used as training and 20% as test data.
Figure 1. Display of image augmentation methods belonging to Mosaic virus class
Table 2. Tomato images and numbers found in Plant Village data set
2.2 CNN
CNN is one of the most popular architectures of deep learning models. The general architecture of CNN is listed as input layer, convolution layers, pooling layers, fully connected layer and an output layer (Softmax). Convolution and pooling layers are used as feature descriptors and fully connected layers are used as classifiers [16-20]. CNN has been widely applied to many tasks such as image recognition, image processing, image detection. Good results have been taken by CNN [18-21]. In this study, ResNet pretrained CNN architectures have been used.
2.3 Resnet architectures
Today, the image classification performance of deep convolutional neural networks is quite good. CNN architectures extract low, medium and high level multi-layered features moreover, the classification performance is quite good. The biggest factor behind this success is convolutional layers. However, the fact that the convolutional layers are high does not always affect the accuracy rate positively. Impaired training accuracy indicates that the system is not easy to optimize. This problem is overcome by the deep residual learning framework found in Microsoft ResNet architectures. Deep Residual Network is similar to networks with convolution, pooling, activation and fully connected layers. The only structure that makes the simple network a permanent network is the identity connection between the layers. Figure 2 shows an example of the Residual block structure.
Figure 2. Example of the residual block structure
The ResNet-18 block is two layers deep, while the ResNet-50 and ResNet-101 blocks are 3 layers deep.
2.4 Neighborhood Component Analysis (NCA) Algorithm
In Neighborhood Component Analysis, it is an algorithm that uses a technique similar to the nearest neighbors to find an area where the points sharing the same label are narrower than the points with different labels. Weights of features in NCA are made using distance measurements. NCA is a non-parametric controlled feature selection method. It is also an alternative distance measurement algorithm to Euclidean distance. The NCA has been developed using K-nearest neighboring algorithm and produces positive weights for each feature [22].
2.5 Cubic Support Vector Machine
The Support Vector Machine (SVM) classification technique is useful in situations with low memory space. SVM has a hyperplane in a multidimensional space that divides classes in the best possible way. Cubic SVM classifier is used in this study. The core function of the classifier is shown in Eq. (1).
$K\left(x_{\mathrm{i}}, x_{\mathrm{j}}\right)=K\left(\mathrm{x}_{\mathrm{i}}^{\mathrm{t}}, \mathrm{X}_{\mathrm{j}}\right)^{3}$ (1)
The proposed method consists of 3 experimental studies. In the first experimental study, the convolutional layers of ResNet-18, ResNet-50 and ResNet-101 architectures have been separately used and 1000 features have been extracted from the last Fully Connected layers. The features obtained from these architectures have been given to the Cubic SVM classifier separately.
Table 3. Methods used in classification of diseases in tomato leaves and success rates
|
Publication and Year |
Approach |
Disease |
Accuracy (in %) |
Training Time |
|
Lu et al., 2018 [2] |
Grey level co-occurrence matrix (GLCM) and ROC curve analysis |
1 Tomato Leaf Diseases |
100 |
- |
|
Brahimi et al., 2017 [3] |
Deep Learning (CNN) |
9 Tomato Leaf Diseases |
99.18 |
Few hours |
|
Karthik et al., 2019 [4] |
Deep Learning (CNN) |
3 Tomato Leaf Diseases |
98 |
10 Hours |
|
Zhang et al., 2018 [5] |
Deep Learning (CNN) |
8 Tomato Leaf Diseases |
97.28 |
44 min 13 sec |
|
Durmus et al., 2017 [11] |
Deep Learning (CNN) |
9 Tomato Leaf Diseases |
97.22 |
- |
|
Suryawati et al., 2018 [6] |
Deep Learning (CNN) |
9 Tomato Leaf Diseases |
95.24 |
- |
|
TM et al., 2018 [7] |
Deep Learning (CNN) |
10 Tomato Leaf Diseases |
95 |
- |
|
Raza et al., 2015 [13] |
Support Vector Machine (SVM) |
1 Tomato Leaf Diseases |
> 90 |
- |
|
Sardogan et al., 2018 [8] |
Deep Learning (CNN) |
4 Tomato Leaf Diseases |
86 |
- |
|
Jia et al., 2017 [9] |
Deep Learning (CNN) |
10 Tomato Leaf Diseases |
88 |
4.5 Hours |
|
Sabrol and Kumar, 2016 [12] |
(FIS), (MLBPNN) |
5 Tomato Leaf Diseases |
87.2 |
- |
|
Fuentes et al., 2017 [10] |
Deep Learning (CNN) |
9 Tomato Leaf Diseases |
83 |
- |
|
Sabrol and Kumar, 2016 [14] |
Decision Tree |
5 Tomato Leaf Diseases: |
78 |
|
|
Our Proposed Method |
Fused CNN, NCA, SVM |
9 Tomato Leaf Diseases: 1 Healty Leaf |
98.2 |
67.96 seconds or 1.3 minutes |
Figure 3. Proposed method
Figure 4. Training time and test accuracy of the proposed method
Table 3 shows the training times and accuracy of each architecture in the SVM classifier. In the second experimental study, the features obtained from three architectures have been combined and given to the Cubic SVM classifier. In the third experimental study, 3000 features obtained have been sorted according to their weights by using NCA algorithm. The first 1000 features with the highest weight have been selected. The selected features have been classified with the Cubic SVM classifier. The visual scheme of the proposed method is given in Figure 3. The proposed method consists of 10 steps. Details of each step of the method are given below:
3.1 Metric criteria for segmentation performance comparison
The features and classification performance rates used in the proposed method are given in Figure 4. As shown in Figure 4, ResNet-50 architecture has the lowest training time, but it has higher than classification performance in ResNet-18 and ResNet-101 architectures. In the experimental study in which all the features have been combined, the performance was achieved by 97%. However, the duration of this study is 156.27 seconds. Although it was about 110 seconds later than the ResNet-50 architecture, it was able to increase its classification performance by only 0.5.
In the proposed method, the first 1000 features with maximum feature weights are used. Figure 4 shows that performance and training time of the proposed method is better than the other methods. In the proposed method, the first 1000 features with maximum feature weights are used. With the proposed method achieved the highest performance with a rate of 98.2%. However, it is seen that the training time is 20 seconds longer than ResNet-101, ResNet-50, ResNet-18 architectures classified using 1000 features.
Figure 5. Confusion matrix
In the confusion matrices are shown in Figure 5, 1 represents healthy class; 2 represents mosaic virus class; 3 represents yellow leaf curl virus class; 4 represents target spot class; 5 represents two spotted spider mite class; 6 represents septoria leaf spot class; 7 represents leaf mold class; 8 represents late blight class; 9 represents early blight class; 10 represents bacterial spot class.
Table 3 shows how many different disease types and the accuracy rates of the studies performed with the same data set.
As shown in Table 3, Lu et. al. (2018) [2] found 100% performance. However, the diseased leaf variety used in this study is only 1 class. Brahimi et al. (2017) [3] found 99.18% success rate with 9 diseased leaf varieties. No clear training time information was provided for this study. However, although the computer used for the proposed method is a highly equipped computer, it is stated that the training period lasts for several hours. When the performance rates and training periods are examined in Table 3, it is seen that our proposed method is in the top 3 with a success rate of 98.2%. The performance of the proposed method is detailed in Table 3. In the proposed method, the performance rate of 98.2% has been achieved by training in only 67.96 seconds. The method proposed in this study has been performed on a laptop with Intel Core i7 - 4510U processor, 8 GB RAM and Windows 10 operating system.
In this article, a new method has been proposed that providing a fast, high accuracy rate for the detection of diseases in tomato leaves. When the studies in the literature have been examined, no other study with such rapid and high performance, which can distinguish 10 tomato classes, has been encountered. ResNet-18, ResNet-50, and ResNet-101 architectures used in the study significantly contribute to classification performance by helping to feature extraction separately. This article's main contribution is to use ResNet architectures as feature extractors to combine the features obtained from these architectures, choose the best distinguishing features with NCA, and establish a very fast tomato disease detection system. The proposed method works at least 40 times faster than studies in the literature. Also, the extraction of features and their classification play an important role in the automatic detection of tomato plant diseases. Experimental results showed that the proposed method detected 98.2% accuracy. The positive effect of using Pretrained CNN models with NCA should not be overlooked. In this study, a hybrid method that can achieve both faster and higher accuracy has been developed. Future studies can be aimed to develop the proposed system further and turn it into a mobile-based application.
[1] Razmjooy, N., Estrela, V.V. (2019). Applications of image processing and soft computing systems in agriculture. IGI Global, 1-337. https://doi.org/10.4018/978-1-5225-8027-0
[2] Lu, J.Z., Zhou, M.C., Gao, Y.W., Jiang, H.Y. (2018). Using hyperspectral imaging to discriminate yellow leaf curl disease in tomato leaves. Precision Agriculture, 19(3): 379-394. https://doi.org/10.1007/s11119-017-9524-7
[3] Brahimi, M., Boukhalfa, K., Moussaoui, A. (2017). Deep learning for tomato diseases: Classification and symptoms visualization. Applied Artificial Intelligence, 31(4): 299-315. https://doi.org/10.1080/08839514.2017.1315516
[4] Karthik, R., Hariharan, M., Anand, S., Mathikshara, P., Johnson, A., R, M. (2020). Attention embedded residual CNN for disease detection in tomato leaves. Applied Soft Computing, 86: 105933. https://doi.org/10.1016/j.asoc.2019.105933
[5] Zhang, K.K., Wu, Q.F., Liu, A.W., Meng, X.Y. (2018). Can deep learning identify tomato leaf disease? Advances in Multimedia, 2018: 1-10. https://doi.org/10.1155/2018/6710865
[6] Suryawati, E., Sustika, R., Yuwana, R.S., Subekti, A., Pardede, H.F. (2018). Deep structured convolutional neural network for tomato diseases detection. 2018 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pp. 385-390. https://doi.org/10.1109/ICACSIS.2018.8618169
[7] Tm, P., Pranathi, A., SaiAshritha, K., Chittaragi, N.B., Koolagudi, S.G. (2018). Tomato leaf disease detection using convolutional neural networks. 2018 Eleventh International Conference on Contemporary Computing (IC3), pp. 1-5. https://doi.org/10.1109/IC3.2018.8530532
[8] Sardogan, M., Tuncer, A., Ozen, Y. (2018). Plant leaf disease detection and classification based on CNN with LVQ algorithm. 2018 3rd International Conference on Computer Science and Engineering (UBMK), pp. 382-385. https://doi.org/10.1109/UBMK.2018.8566635
[9] Jia, S., Jia, P., Hu, S., Liu, H. (2017). Automatic detection of tomato diseases and pests based on leaf images. 2017 Chinese Automation Congress (CAC), pp. 2537-2510. https://doi.org/10.1109/CAC.2017.8243388
[10] Fuentes, A., Yoon, S., Kim, S.C., Park, D.S. (2017). A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors, 17(9): 2022. https://doi.org/10.3390/s17092022
[11] Durmuş, H., Güneş, E.O., Kırcı, M. (2017). Disease detection on the leaves of the tomato plants by using deep learning. 2017 6th International Conference on Agro-Geoinformatics, Fairfax, VA, pp. 1-5. https://doi.org/10.1109/Agro-Geoinformatics.2017.8047016
[12] Sabrol, H., Kumar, S. (2016). Fuzzy and neural network based tomato plant disease classification using natural outdoor images. Indian Journal of Science and Technology, 9(44): 1-8. https://doi.org/10.17485/ijst/2016/v9i44/92825
[13] Raza, S., Prince, G., Clarkson, J.P., Rajpoot, N.M. (2015). Automatic detection of diseased tomato plants using thermal and stereo visible light images. PLoS ONE, 10(4): e0123262. https://doi.org/10.1371/journal.pone.0123262
[14] Sabrol, H., Kumar, S. (2016). Intensity based feature extraction for tomato plant disease recognition by classification using decision tree. International Journal of Computer Science and Information Security (IJCSIS), 14(9): 622-626.
[15] Hughes, D., Salathé, M. (2015). An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint arXiv:1511.08060.
[16] Song, X., Gao, S., Chen, C., Wang, S. (2020). A novel face recognition algorithm for imbalanced small samples. Traitement du Signal, 37(3): 425-432. https://doi.org/10.18280/ts.370309
[17] Tuncer, T., Dogan, S., Özyurt, F., Belhaouari, S.B., Bensmail, H. (2020). Novel multi center and threshold ternary pattern based method for disease detection method using voice. IEEE Access, 8: 84532-84540. https://doi.org/10.1109/ACCESS.2020.2992641
[18] Ozyurt, F., Tuncer, T., Subasi, A. (2021). An automated COVID-19 detection based on fused dynamic exemplar pyramid feature extraction and hybrid feature selection using deep learning. Computers in Biology and Medicine, 132: 104356. https://doi.org/10.1016/j.compbiomed.2021.104356
[19] Diker, A., Sönmez, Y., Özyurt, F., Avcı, E., Avcı, D. (2021). Examination of the ECG signal classification technique DEA-ELM using deep convolutional neural network features. Multimedia Tools and Applications, 83: 24777-24800. https://doi.org/10.1007/s11042-021-10517-8
[20] Özyurt, F. (2021). Automatic detection of COVID-19 disease by using transfer learning of light weight deep learning model. Traitement du Signal, 38(1): 147-153. https://doi.org/10.18280/ts.380115
[21] Özyurt, F., Avcı, E., Sert, E. (2020). UC-merced image classification with CNN feature reduction using wavelet entropy optimized with genetic algorithm. Traitement du Signal, 37(3): 347-353. https://doi.org/10.18280/ts.370301
[22] Özyurt, F. (2019). Uzaktan Algılama Görüntülerinin Evrişimsel Sinir Ağları ve Komşuluk Bileşen Analizi Tabanlı Özniteliklerinin Sınıflandırılması. Afyon Kocatepe Üniversitesi Fen Ve Mühendislik Bilimleri Dergisi, 19(3): 669-675. https://doi.org/10.35414/akufemubid.558311
