Velpula Sekhara Babu* | Nidumolu Venkat Ram
© 2022 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Weeding is the fundamental task in agriculture to increase yields crop. Accurate weed recognition is major prerequisite in precision agriculture. Precision weeding significant reduces the usage of herbicides in farming. Deep learning has been a major endeavor for enhancing the learning performance, particularly for classification. This paper proposes a Deep Residual Convolutional neural network (DRCNN) with Contrast Limited Adaptive Histogram Equalization (CLAHE) for weed and crop classification helpful for accurate individual targeting of weeds. In this method, initially data augmentation is performed to avoid overfitting on training data, A deeper residual network architecture is defined through residual connections in CNN architecture this architecture improves gradient flow through the network and for training the deeper network. The experiments are carried out on the publicly available dataset with four groups of images viz., soil, grass, soybean and broadleaf. Different state-of-the-art pretrained networks like AlexNet, and VGG-16 Net are also investigated and the results are compared. The proposed method yielded an accuracy of 97.3% which is superior to other methods.
CLAHE, classification, CNN, data augmentation, deep residual weed and crop
Crop production is the major objective of agriculture. For global food management it is very essential. Crop is a cultivated plant while weed is an unwanted plant which grows randomly in the field. Weed control is very vital in crop production. Precision weed control is a challenging task to get quality crops, and can reduce the amount of herbicides to ensure the best results. This is possible with the use of latest techniques based on artificial intelligence. Different machine learning and deep learning algorithms are used recently in precision agriculture. There can be many varieties of weed that are harmful for crop production and need to be detected in the early stages of growth. The basic resources which are essential for crop growth viz., soil, water, sunlight and also the fertilizers used are effected by these unwanted weeds. From the literature it is observed that about one third of crops are effected due to growth of weeds. Weed and crop classification is fundamental in agriculture, rural medicine, forestry and commercial applications. In the past, weed detection and removal was done by men, later some research work was done and implemented on weed–crop classification for automatic detection and removal. subsequently robots came into existence, but suffer accuracy problem existing with traditional classification techniques. So we propose a deep learning method to perform weed/crop classification.
Weed and crop recognition is a challenging research area and various computer vision algorithms are developed focusing this area. The fundamental approaches are feature based and recently different machine learning and deep learning algorithms are developed aiming at recognizing weeds accurately. Contrast-Limited-Adaptive-Histogram- Equalization (CLAHE) is widely used image enhancement method. The following subsections outlines the related work of the preprocessing method, Feature based and Deep learning based approaches.
2.1 Feature-based approaches
Earlier, weed and crop identification methods are based on low level feature extraction like color, texture and shape. In research Shape, texture and fractal dimensions are integrated for weed recognition. In their work Support vector machine and Shafer-Dempster evidence theory are used [1]. In maize and weed classification Color indices is used. Here Support Vector Data Description (SVDD) is also used and investigated under different weather conditions [2]. In precision farming weed control task is vital. To achieve this SURF features integrated with an optimized Gaussian Mixture Model is employed [3]. Plant and weed classification in UAV imagery is done for smart farming [4]. In addition, multi spectral features are used in weed recognition [5]. The features mentioned used for recognition are taken on human experience. The recognition performance is different on different datasets [6].
2.2 Deep learning based approaches
With feature based methods, different machine learning techniques are used for classification purpose. Recently these are replaced by deep learning techniques to detect weed in real time. Deep learning gives solution for many drawbacks in traditional methods and reduces the human effort in feature extraction from agriculture images [7, 8]. As example Sugar beet plant and background separation with semantic segmentation for RGB data by convolution neural networks in real time fields was performed [9]. Real time blob wise Sugar beet and weed classification was also implemented by using CNN [10]. Different crop like Maize plant classification from weed using CNN with different filters and hardware also yielded the best results compared to traditional methods [11]. Leaf species detection with smart phone was implemented based on CNN approaches by extracting the features like shape, texture, color., etc. [12] and two CNN approaches, one approach for plant- soil classification and other approach for crop-weed classification with segmentation [13], weed detection with deep CNN [14] is done. In precision agriculture sliding window approach for the detection of weed regions with CNN [15]. Convolvulus sepium detection in sugar beet field is done using D-CNN based on YOLOv3 architecture [16]. For multiclass weed classification for deep weeds dataset inception-v3 and ResNet-50 are used [17]. Weed detection for canola fields is implemented by deep CNN along with maximum likelihood classification [18]. Detection of weed instances in cereal fields is done with fully CNN based on a modified version of the GoogLe Net architecture. It is used to detect weeds in winter wheat fields particularly with heavy leaf occlusion [19]. Generally in CNN random initial weights are used. Instead of this process, K-means unsupervised feature learning method is used in CNN [20]. Combination of multiple features and deep confidence network is used for weed detection [21]. Unsupervised deep learning and semi-automatic data labeling is used for weed identification which yielded an accuracy of about 97% [22].
These Deep learning approaches yield good accuracy when large dataset is available for training. So these are not suitable to agricultural recognition as the labeled data available is limited [23]. CNN feature based graph convolutional network (GCN) based on extracted weed CNN features for four different weed datasets with different methods viz., AlexNet, VGG16 and ResNet-101 are investigated [24]. The design of an accurate weed and crop recognition model with significant learning capability when the availability of data is very limited is still a challenge. In this paper we focused on robust and high efficient crop/weed classification system using convolutional neural networks.
2.3 Preprocessing
There are different Histogram Equalization methods proposed in research for image enhancement. In Dynamic Histogram Equalization the histogram of image is partitioned depending on local minima. Then partitioning is done considering different gray level ranges. This is done before equalization [25]. In Brightness Protecting Dynamic Histogram Equalization (BPDHE) mean brightness of the image is maintained. The mean intensity of both the output image and the input image is nearly the same [26]. However CLAHE has superior contrast limiting than ordinary adaptive histogram equalization. CLAHE is widely used for enhancing medical imagery, satellite images, etc. [27].
3.1 Dataset
The data for this work is taken from a public repository [28]. As per dos Santos Ferreira (2017), soybean crop is grown in Brazil, the images are captured using drone, the size of this dataset is 1.2GB. These captured images contain about 15336 images of the soil, soybean, broadleaf and grass weeds. In these images 3249 belong to soil, 7376 images are of soybean, 3520 are of grass and 1191images are of broadleaf weeds. In this work, from this dataset 1000 images from each group are taken. Figure 1 shows Examples of image dataset classes.
Figure 1. Examples of image dataset classes. First, Second, Third and Fourth Columns examples of Broadleaf, Grass, Soil, Soybean classes
3.2 Contrast limited adaptive histogram equalization
As a preprocessing step, image enhancement through CLAHE is used for better visual interpretation of images.
CLAHE algorithm improves the contrast of image and limits the amplification, for improving the quality of image. In conventional adaptive histogram equalization the noise in the near constant regions in image is amplified. this is due to over amplification of contrast in the near constant regions. This problem is addressed by CLAHE by limiting the contrast. CLAHE algorithm has basically three steps i) portition of the input image ii) For each part histogrm equalization is done using a predefined threshold iii) The histogram is computed for every partition as set of bins. If these values in a bin are above the threshold, they are distributed to the remaining bins. Following this the Cumulative Distribution Function value is computed for the histogram values. These Cumulative Distribution Function values of each part are scaled. Then mapped using the input image pixel values. With bilinear interpolation, the resulting blocks are combined to generate an output image with improved contrast [29].
3.3 Architecture of DRCNN
Deep residual convolutional neural network architecture [30, 31] is used in Figure 2. architecture and specifications. The propagation of Parameter gradients will become easy from the final layer to the previous layers of the network because of residual connections. Consequently, the training of deeper networks is possible and highly accurate results on very difficult tasks are achieved.
Figure 2. Architecture deep residual convolutional neural network
3.4 Data augmentation
The batches of data are transformed with operations like resizing, rotation, and reflection. During training, in a random manner the images are flipped. This is done along the vertical axis. Then the images are translated to four pixels in horizontal direction and vertical direction. Data augmentation of training image data with the above operations will avoid over fitting of the network. Here at each epoch, the training data is perturbed. Thus different data set is used at each epoch, while the number of images at each epoch does not change.
3.5 Proposed network architecture
Figure 3. Proposed architecture
Figure 3 shows the architecture of residual network consists of i) Main branch having sequential connection of convolutional layer l, batch normalization layer, and ReLU layers ii) Additionally the main branch convolutional units are bypassed with Shortcut connections. The outputs of these shortcut connections and convolutional blocks are added element by element. The Shortcut connections makes the flow of the parameter gradients from the output layer to the previous layers of the network to be easy, thus making network training faster and more reliable.
3.6 Main branch
There are five sections in the main branch: i) Image input layer, ii) 3 stages of convolutional layers with different feature sizes viz., 32x32, 16x16, and 8x8, respectively, with each stage containing 2 convolutional units, iii) The last section contains global average pooling, fully connected layer, softmax layer, and classification layer.
3.7 Shortcut connections
Shortcut connections are added around the convolutional units. No operations are performed by most of the shortcut connections. They just add element wise to the convolutional units outputs. A deeper residual network is created, three convolutional units per each stage i.e., 9 convolutional units and with a width of 16.
3.8 Training of network & evaluation
To train the network, different training options are specified. Here stochastic gradient descent with momentum optimizer is used. Gradient descent method minimizes the objective function $\mathrm{J}(\theta)$, which is a function of model parameter $\theta$. This is done by updating the parameters in the opposite direction of the gradient of the objective function $\nabla_{\theta} \mathrm{J}(\theta)$. To achieve this local minimum, the learning rate $\eta$ is selected properly. This selection determines the step size. $\gamma$ is the momentum term. v represents the weights of the neural network. Updation of weights are done by the formula
$v_{t}=\gamma v_{t-1}+\eta \Delta_{\theta} \mathrm{J}(\theta)$
The network is trained for 50 epochs. The accuracy on the training set and also on the validation set is obtained.
For the evaluation of results on both training images and validation images a confusion matrix is obtained. This is a table which illustrates the details of actual and predicted classes generated by the system. This matrix represents the actual in rows and predicted in columns. Diagonal elements indicate number of correctly classified images. The other elements indicate misclassified images. Figure 5 and Figure 6 shows the confusion matrix of size 4x4 for the 4 categories broad leaf, grass, soil, soybean images classes using DRCNN without and with CLAHE algorithm.
Results
Validation Accuracy: 97.25%
Elapsed time: 17 minutes 44 seconds
Iteration per Epoch: 87
Maximum Iterations: 4350
Hardware source: Single CPU
Learning rate: 0.025
Figure 4. Training progress with CLAHE
From Figure 4, the graph is of two parts one is Iteration(x-axis) vs Accuracy(y-axis).in the upper x-axis 10, 20, 30, 40, 50 are epochs and lower divisions of 0, 500, 1000, 1500, … 4500 are Iterations and y-axis from 0 to 100 in steps of 10. Similarly in the second part of the graph is for Iteration(x-axis) vs Loss(y-axis).in the x-axis upper divisions of 10, 20, 30, 40, 50 are epochs and lower divisions of 0, 500, 1000, 1500, … are Iterations, and y-axis from 0 to 0.14 in steps of 0.2.
Figure 5. Confusion matrix for weed/crop classification with DRCNN
Figure 6. Confusion matrix for weed crop classification with DRCNN + CLAHE
Figure 7. Classification of each group of images in four groups
From Figure 7, it is observed that 289 broableaf, 291 grass, soil 300 and 287 soybean are correctly classified out of 300 in each class of images.
Figure 8. Classification accuracy of individual groups of images and over all accuracy
Table 1. Performance comparison
|
S.No |
Method /Network |
Accuracy(%) |
|
1 |
Alex-Net |
95.17 |
|
2 |
VGG-16 Net |
95.75 |
|
3 |
DRCNN |
96.33 |
|
4 |
DRCNN with CLAHE (Proposed) |
97.25 |
Figure 9. Comparison with other pre-trained networks
Figure 5 and Figure 6 are the confusion matrices for weed/crop classification with DRCNN and DRCNN + CLAHE respectively. From Figure 7. It is observed that 289 broableaf, 291 grass, soil 300 and 287 soybean are correctly classified out of 300 in each class of images. Figure 8 shows the classification performance of broadleaf, grass, soil, soybean and average performance of these 4 classes.
Table 1 shows the performance comparison of different methods Alex-Net, VGG-16 Net with DRCNN and DRCNN with CLAHE(Proposed). Figure 9 is the graphical representation of Table 1.
In this work, a method was developed to classify different types of weeds and soybean crop images, which would be helpful in discriminating between different types of weeds and also to detect weed images in crops. Deep Residual Convolutional neural network (DRCNN) worked well in recognizing the images, with yielded accuracy of 96.3% after applying simple histogram equalization for enhancing the images. However, with the CLAHE algorithm, the classification was much better with an accuracy of 97.3%. Data augmentation is additional feature which enhanced the recognition capability of CNN architecture. Compared to pretrained architectures viz., Alexnet and VGG-16, our method demonstrated superior performance. In future using the designed architecture, the detection of weeds in soybean crop images can be investigated.
[1] Bakhshipour, A., Jafari, A. (2018). Evaluation of support vector machine and artificial neural networks in weed detection using shape features. Computers and Electronics in Agriculture, 145: 153-160. https://doi.org/10.1016/j.compag.2017.12.032
[2] Zheng, Y., Zhu, Q., Huang, M., Guo, Y., Qin, J. (2017). Maize and weed classification using color indices with support vector data description in outdoor fields. Comput. Electron. Agric., 141: 215-222. https://doi.org/10.1016/j.compag.2017.07.028
[3] Kounalakis, T., Triantafyllidis, G.A., Nalpantidis, L. (2018). Image-based recognition framework for robotic weed control systems. Multimedia Tools and Applications, 77(8): 9567-9594. https://doi.org/10.1007/s11042-017-5337-y
[4] Lottes, P., Khanna, R., Pfeifer, J., Siegwart, R., Stachniss, C. (2017). UAV-based crop and weed classification for smart farming. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3024-3031. https://doi.org/10.1109/ICRA.2017.7989347
[5] Sa, I., Chen, Z., Popović, M., Khanna, R., Liebisch, F., Nieto, J., Siegwart, R. (2017). Weednet: Dense semantic weed classification using multispectral images and MAV for smart farming. IEEE Robotics and Automation Letters, 3(1): 588-595. https://doi.org/10.1109/LRA.2017.2774979
[6] Bah, M.D., Hafiane, A., Canals, R. (2018). Deep learning with unsupervised data labeling for weed detection in line crops in UAV images. Remote Sensing, 10(11): 1690. https://doi.org/10.3390/rs10111690
[7] Liu, B., Bruch, R. (2020). Weed detection for selective spraying: a review. Current Robotics Reports, 1(1): 19-26. https://doi.org/10.1007/s43154-020-00001-w
[8] Yang, X., Sun, M. (2019). A survey on deep learning in crop planting. In IOP Conference Series: Materials Science and Engineering, 490(6): 062053. https://doi.org/10.1088/1757-899X/490/6/062053
[9] Milioto, A., Lottes, P., Stachniss, C. (2018). Real-time semantic segmentation of crop and weed for precision agriculture robots leveraging background knowledge in CNNs. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2229-2235. https://doi.org/10.1109/ICRA.2018.8460962
[10] Milioto, A., Lottes, P., Stachniss, C. (2017). Real-time blob-wise sugar beets vs weeds classification for monitoring fields using convolutional neural networks. ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences, 41-48. https://doi.org/10.5194/isprs-annals-IV-2-W3-41-2017
[11] Andrea, C.C., Daniel, B.B.M., Misael, J.B.J. (2017). Precise weed and maize classification through convolutional neuronal networks. In 2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM), pp. 1-6. https://doi.org/10.1109/ETCM.2017.8247469
[12] Vilasini, M. (2020). The CNN approaches for classification of Indian leaf species using smartphones. Computers, Materials & Continua, 62(3): 1445-1472.
[13] Fawakherji, M., Youssef, A., Bloisi, D., Pretto, A., Nardi, D. (2019). Crop and weeds classification for precision agriculture using context-independent pixel-wise segmentation. In 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 146-152. https://doi.org/10.1109/IRC.2019.00029
[14] Yu, J., Schumann, A.W., Cao, Z., Sharpe, S.M., Boyd, N.S. (2019). Weed detection in perennial ryegrass with deep learning convolutional neural network. Frontiers in Plant Science, 10: 1422. https://doi.org/10.3389/fpls.2019.01422
[15] Kantipudi, K., Lai, C., Min, C.H., Chiang, R.C. (2018). Weed detection among crops by convolutional neural networks with sliding windows. In 14th International Conference on Precision Agriculture, Quebec.
[16] Gao, J., French, A.P., Pound, M.P., He, Y., Pridmore, T.P., Pieters, J.G. (2020). Deep convolutional neural networks for image-based Convolvulus sepium detection in sugar beet fields. Plant Methods, 16(1): 1-12. https://doi.org/10.1186/s13007-020-00570-z
[17] Olsen, A., Konovalov, D.A., Philippa, B., Ridd, P., Wood, J.C., Johns, J., White, R.D. (2019). DeepWeeds: A multiclass weed species image dataset for deep learning. Scientific Reports, 9(1): 1-12. https://doi.org/10.1038/s41598-018-38343-3
[18] Asad, M.H., Bais, A. (2020). Weed detection in canola fields using maximum likelihood classification and deep convolutional neural network. Information Processing in Agriculture, 7(4): 535-545. https://doi.org/10.1016/j.inpa.2019.12.002
[19] Dyrmann, M., Jørgensen, R.N., Midtiby, H.S. (2017). RoboWeedSupport-Detection of weed locations in leaf occluded cereal crops using a fully convolutional neural network. Advances in Animal Biosciences, 8(2): 842-847. https://doi.org/10.1017/S2040470017000206
[20] Tang, J., Wang, D., Zhang, Z., He, L., Xin, J., Xu, Y. (2017). Weed identification based on K-means feature learning combined with convolutional neural network. Computers and Electronics in Agriculture, 135: 63-70. https://doi.org/10.1016/j.compag.2017.01.001
[21] Deng, X., Qi, L., Ma, X., Jiang, Y., Chen, X., Liu, H., Chen, W. (2018). Recognition of weeds at seedling stage in paddy fields using multi-feature fusion and deep belief networks. Transactions of the Chinese Society of Agricultural Engineering, 34(14): 165-172. https://doi.org/10.11975/j.issn.1002-6819.2018.14.021
[22] dos Santos Ferreira, A., Freitas, D.M., da Silva, G.G., Pistori, H., Folhes, M.T. (2019). Unsupervised deep learning and semi-automatic data labeling in weed discrimination. Computers and Electronics in Agriculture, 165: 104963. https://doi.org/10.1016/j.compag.2019.104963
[23] Wang, A., Zhang, W., Wei, X. (2019). A review on weed detection using ground-based machine vision and image processing techniques. Computers and Electronics in Agriculture, 158: 226-240. https://doi.org/10.1016/j.compag.2019.02.005
[24] Jiang, H., Zhang, C., Qiao, Y., Zhang, Z., Zhang, W., Song, C. (2020). CNN feature based graph convolutional network for weed and crop recognition in smart farming. Computers and Electronics in Agriculture, 174: 105450. https://doi.org/10.1016/j.compag.2020.105450
[25] Abdullah-Al-Wadud, M., Kabir, M.H., Dewan, M.A.A., Chae, O. (2007). A dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, 53(2): 593-600. https://doi.org/10.1109/TCE.2007.381734
[26] Ibrahim, H., Kong, N.S.P. (2007). Brightness preserving dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, 53(4): 1752-1758. https://doi.org/10.1109/TCE.2007.4429280
[27] Asha, S., Sreenivasulu, G. (2018). Satellite image enhancement using contrast limited adaptive histogram equalization. Int. J. Sci. Res. Sci. Eng. Technol, 4(1): 1070-1075.
[28] dos Santos Ferreira, A., Freitas, D.M., da Silva, G.G., Pistori, H., Folhes, M.T. (2017). Weed detection in soybean crops using ConvNets. Computers and Electronics in Agriculture, 143(C): 314-324. https://doi.org/10.1016/j.compag.2017.10.027
[29] Dubey, U., Chaurasiya, R.K. (2021). Efficient traffic sign recognition using CLAHE-based image enhancement and ResNet CNN architectures. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 15(4): 1-19. https://doi.org/10.4018/IJCINI.295811
[30] Krizhevsky, A., Hinton, G. (2009). Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4).
[31] He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778. https://doi.org/10.1109/CVPR.2016.90
