Efficient Method Using Attention Based Convolutional Neural Networks for Ceramic Tiles Defect Classification

Traditional defect detection algorithms have not achieved stable results despite a strong dependence on illumination, disk grain structure, defect morphology, and defect size [2]. On the other hand, several studies have been presented to achieve high accuracy for defect detection on ceramic tiles [3-9]. Ragab and Alsharay [3] have proposed a solution consisting of two steps to minimize the time required for detecting defects in ceramic images. In the first step, it detects defects in each of the eight parts of the image, and in the second, it determines their type using the algorithms of defects of spots and cracks. experiments show that the time of detection and classification of defects in ceramic tiles is reduced compared to previous work. However, they are used a limited number of images and it is limited to too low detection rate. Hanzaei et al. [4] used processing techniques such as median filter, local variance rotation invariant measure, thresholding, and morphological closure operations, and then use a support vector machine (SVM) for multiclass classification. This solution improves the automatic detection rate of cracks and holes on ceramic tile surfaces by accurately detecting edge and defect regions and eliminating unimportant areas. However, this work does not deal with tile’s textured surfaces. While Chen et al. [5] used a decision tree algorithm to classify the defects of textured ceramic tiles based on a Fourier transformer to remove the background, they applied Laplacian sharpening, histogram specification, and median filtering. They then extracted the features using Lavel Co-occurrence Matrix (GLCM) and Casagrande et al. [6] proposed a hybrid algorithm to extract features from smooth and textured ceramic tile images, a combination of segmentation-based fractal texture analysis and discrete wavelet transform methods was used after applying the preprocessing techniques, which were then introduced for the five classifiers. Within them, the SVM classifier reaches 99.01% for smooth images and 97.89% for textured images. This solution only deals with the defects of the pattern's discontinuity despite using textured tiles. In addition, it is limited to a single model of the pattern of textured ceramic tiles. Mariyadi et al. [7] proposed to use GLCM to extract fourteen properties of the defect area, which are then sent to the artificial neural network for classification. The defect area was clipped from the image after counting the mean hue, deviation square and euclidean distance, threshold segmentation, and morphological operation. Despite the limited number of images without texture, there are errors in the classification of defects, especially the pinholes which classify as crack. Zorić et al. [8] developed a method for detecting defects on the surfaces of biscuit tiles which based on Fourier Spectrum for extraction features of defects and on K-nearest neighbors (KNN) and random forest for classification. Zhang et al. [9] proposed an efficient method is proposed based on three basic steps: image preprocessing, defect detection, and defect determination. The authors remove the background information by SRR algorithm, adjust the lateral contrast, and then apply the spatial distribution variance (CSDA) and the color spot area weight (CSAW) to the HSV color. A defect saliency map is generated, and the defect area is divided into blocks according to the boundaries of the defect rectangle to extract the feature vector color, which is then fed to the SVM algorithm. The method achieves good performance with an accuracy rate of 98.75%. The performance of this method decreases with the increase in the complexity and variety of the textures of the tile surfaces, due to the increase in the number of characteristics of the colors to be represented.

However, recent contributions based on deep learning models have appeared. A new method, "Segmentation Model for Cracks in Ceramics (CCS)," is proposed [10]. Indeed, the latter is based on the U-net model and a combination of pre-processing techniques that are applied to the images to render prominent cracks in a white and perfect font based on morphological and correction operations. The accuracy of defect detection is very high, reaching 99.9%. However, the effectiveness of this solution has been tested on a limited number of ceramic tile images and does not treat just one defect. Nogay et al. [11] used the AlexNet pre-trained model to classify invisible cracks in ceramic tile images based on acoustic noise. Indeed, a dataset of nine classes of invisible crack images of different sizes but similar structural properties are introduced. The model achieves better accuracy in detecting cracks. The model achieves better accuracy in detecting cracks despite the generalization supported by this work. However, it does not specify the location of cracks, which is essential for automated inspection of cracks, and it does not remove noise in images. Thus, a limited number of images are used for training and testing. While Stephen et al. [12] proposed a lightweight convolutional neural network to automate the detection of cracks in ceramic tiles with smooth surfaces. Feature extraction and classification are performed in parallel in this CNN model. Indeed a better accuracy in detecting cracks is obtained.

4.png

We introduce a new dataset for defect detection of ceramic tiles. The collected images are 1032 RGB images. These images are of different shapes (rectangular and square) and appearances (texture, illumination). The collected images are divided into two classes defect and no defect. Figure 5 below shows samples of the collected defect-free and defect images.

5.png

Figure 5. Examples of images from our dataset

4.3 Preprocessing

The preprocessing step of our model consists of two phases. The first is to remove the background from the image, and the second phase is to create a mask for the defective images to better represent the position and shape of the defects and reduce the complexity of detecting it on the ceramic tiles. In the automation of the industrial inspection system, the great challenge is detecting the different types of defects in the texture surfaces [23]. Indeed, the specificity of each defect on a ceramic requires a different treatment for the creation of the mask. The purpose of adding these masks is to improve the representation of features. In our solution, we have adopted three types of masks for the images of the defective tiles: the first two are highlighted the breaks of the corners of the ceramic tiles and the pinholes via the segmentation and morphological operations. Thus, one extracts the white parts, which appeared because of the discontinuity of the textures.

4.3.1 Corner and pinholes masks

This step consists in creating a binarized image, where the pixels of the latter are composed of two black and white values. Then, from a grayscale image, we use the Otsu function [24] with the operation of inversion to identify the tile in white and the background and the pinholes in black. The Otsu method created by Nobuyuki Otsu which is an automatic thresholding method that is based on the search in the histogram of the image on the optimal value among the thresholding values denoted (t) which is in the interval [0, 255]. Indeed, this is calculated with maximize the variance inter-classes, denoted by $\left(\sigma_y^2\right)$ and minimize the variance intra-classes denoted by $\left(\sigma_w^2\right)$ for separate the pixel values of the background (bg) and foreground (fg). These are calculated by following equations:

$\sigma_w^2(t)=\omega_{b g}(t) \sigma_{b g}^2(t)+\omega_{f g}(t) \sigma_{f g}^2(t)$          (4)

$\sigma_y^2=\sigma^2-\sigma_w^2$            (5)

$\omega_{b g}=\sum_{i=0}^t P(i)$           (6)

$\omega_{f g}=\sum_{i=t+1}^{255} P(i)$           (7)

Such as:

ω_bg: presents the sum of the probabilities of the pixels P(i) which are lower than the threshold t devised by the overall probability of all pixels of an image.

ω_fg: presents the sum of the probabilities of the pixels P(i) greater than the threshold t divided by the overall probability of all an image's pixels.

where: the probability of pixels is calculated by this equation:

$P(i)=\frac{n_i}{n}$          (8)

n_i: the value of the i^eme pixelin image.

n: the average value of all pixels of the background and foreground classes.

$\sigma_{\mathrm{bg}}^2$: presents the variance of background class and $\sigma_{\mathrm{fg}}^2$: presents the variance of foreground class, where are calculated:

$\sigma_{b g}^2=\frac{\sum_{i=0}^t \, \, \left(N 1(i)-M o y_{b g} \, (T)\right) \, \, ^2 \, \, \times P(i)}{\omega_{b g}}$          (9)

$\sigma_{b f}^2=\frac{\sum_{i=t+1}^{255} \, \, \, \, \left(N 2(i)-M o y_{b f}(T)\right) \, \, ^2 \, \, \times P(i)}{\omega_{b f}}$            (10)

where:

N1: is a vector from 0 to t-1.

N2: is a vector from t to 255.

Moy_bg: the average of the background class and Moy_fg: the average of the foreground class, which are calculated by the following equations:

$\operatorname{Moy}_{b g}(t)=\frac{\sum_{i=0}^t  \, \, \, \, N 1(i) \times P(i)}{\omega_{b g} \,(t)}$           (11)

$\operatorname{Moy}_{f g}(t)=\frac{\sum_{i=t+1}^{255} \, \, \, \, N 2(i) \times P(i)}{\omega_{f g} \, (t)}$            (12)

Finally, we remove the noise from the binary images by operating the morphological opening with a filter of a size 3×3 filled by ones to obtain a clear image, followed by a dilation operation. Figure 6 shows a resulting pinholes and corner mask after performing this series of operations on two faulty images.

6.png

Figure 6. The steps to generate a mask of two images with a default: (a) Input image, (b) Gray scale, (c) Binary image, (d) Opening morphology and (e) Mask image

7.png

Figure 7. Defective images with the locations of the pattern discontinuity by red rectangles

4.3.2 Texture mask

In order to obtain a more precise representation of the discontinuity of patterns on the surfaces of ceramic tiles like the examples shown in Figure 7, we used the HSV (Hue Saturation Value) color mode [25], which Alvy Ray Smith creates. This model is a system of human visual perception consisting of three components: Hue (color), Saturation (intensity), and Value (brightness). Studies use the HSV color mode to improve color defects in colored images, such as detecting defects in fruit [26] and on the surface of printing rollers [27].

For conversion between RGB mode to HSV mode, the three values: Red (R), Green (G), and Blue (B), are used to calculate the H, S, and V values according to the following functions knowing that: $R, G, B \in[0,1]$.

$M a x=\max (R, G, B)$          (13)

$\operatorname{Min}=\min (R, G, B)$           (14)

$H=\left\{\begin{array}{l}0, \text { if } R=G=B \\ 60^{\circ} \times\left(0+\frac{G-B}{\text { Max-Min }}\right), \text { if Max }=R \\ 60^{\circ} \times\left(2+\frac{B-R}{\text { Max-Min }}\right), \text { if Max }=G \\ 60^{\circ} \times\left(4+\frac{R-G}{\text { Max-Min }}\right), \text { if Max }=B\end{array}\right.$           (15)

$S= \begin{cases}0, & \text { if } R=G=B \\ \frac{\text { Max-Min }}{\text { Max }}, & \text { else }\end{cases}$            (16)

V=Max          (17)

An example of the texture mask is shown in Figure 8.

8.png

Figure 8. Generation of a texture mask (c) from the input image (a) HSV mode (b)

4.4 Classification

Our ceramic tile classification model has an architecture that combines a convolutional neural network (CNN) and an attention mechanism. The CNN is composed of four convolutional layers, two first consequential layers to extract the low level characteristics, followed by a ReLU activation function and a Maxpooling layer. Then the CBAM layer followed by two convolutional layers. After each layer a ReLU function is applied. The second CBAM attention module is inserted. We adopted a regularization using the dropout by a factor of 0.25 to avoid overfitting. Finally, a fully connected (FC) layer with 100 neurons followed by a sigmoid activation function to classify the input image whose dimension is 210×210×3 as defective or not.

In this section, we present the results obtained by our model. Indeed, an evaluation of the effect of the parameters chosen for our model, a quantitative comparison with state-of-the-art methods, and finally, an ablation study is conducted to show the efficiency of each component of our model.

6.1 Performance evaluation of setting parameters of CBAM module

For the CBAM attention module, we tested the performance of the variant configurations for more efficiency in detecting faults. In fact, experiments are menu by varying the values of the reduction ratio of the channel module.

6.1.1 The Ratio reduction (R)

Table 4 presents the results of experiments comparing the accuracy, precision, and recall of the classification obtained by our model with the values 8, 16, and 32 assigned to the ratio of the channel module reduction. In this study, we have kept the default filter size of the convolutional layer of the spatial module, which is 7.

All accuracies achieved by the three combinations are satisfactory. Perfect accuracy is reached by minimizing the dimensions of the feature maps by the ratio 16, while for the values 32 and 8, a decrease in the accuracy is observed by 0.07% and 0.14%, respectively. Indeed, the exact choice of the ratio considerably improves the relevant features generated in the channel attention maps. Consequently, better performances of defects detection are achieved.

Table 4. Comparison of classification accuracies according to Channel module reduction ration values

6.1.2 The impact of the number and position of CBAM module

Table 5 illustrates the performance evaluation of our model as a function of the position and number of the CBAM module in our CNN model. Indeed, a variation of accuracy, precision, and recall are analyzed to choose the optimal configuration to obtain an efficient extraction of discriminative features.

We note from the results shown in Table 5 that the integration of the two CBAM modules after the conv 2 and conv 4 layers, respectively, produces good defect classification performance in terms of accuracy of 99.93%, precision of 100%, and recall of 99.92%. In addition, the extraction of relevant features after conv1 by the CBAM module leads to a significant classification of defects by an accuracy of 99.86%, precision of 99.92%, recall of 99.92%.

Table 5. The effect of position and number of the CBAM attention module on the performance of our model

In this section, we will discuss the choice of parameters of our proposed model to effectively classify tile images, including the learning rate and classification type.

6.1.3 The impact of learning rate values

9.png

Figure 9. The effect of learning rate on the classification accuracy of our CNN model

Figure 9 shows the accuracy of the validation for various learning rates. The validation accuracy decreases when the learning rate is too high 0.01. When the learning rate is 0.01, the best performance is attained. The accuracy value reaches 99.93%.

6.1.4 Effect of type of classification

Table 6 shows the effect of the type of classification (binary or multiclass) on the performance of our model in classifying the images of ceramic tiles. Indeed, we used two datasets to prove the efficiency of our model. Dataset 1 is a binary dataset where the tiles are organized in two classes, defect, and non defect, while dataset 2 is a multiclass dataset where the images of the non defect class are divided into three distributions corresponding to the three types of defects taken into consideration by our study: with corner breakage, pinholes, and pattern discontinuity.

The three evaluation metrics, accuracy, precision, and recall, are used to measure the performance of our model for a binary classification on dataset 1 and multiclass on dataset 2.

Table 6 illustrates the comparison results among binary classification and multi-classification tasks. These results demonstrated a significant distinction between our model's binary and multiclass classification. Compared to a multiclass classification, where each type of defect is represented by a different class, our model generalizes more effectively for the binary classification of ceramic tiles. Our model performs better with a binary classification, increasing accuracy, precision, and recall by 4.18%, 4.09%, and 0.82% respectively.

Table 6. Classification performance of our model on different datasets

6.2 Effect of masks

Augmenting our dataset with mask images in the preprocessing part plays a significant role in improving the performance of our solution. Indeed, to evaluate the effect of adding the masks to the defect images on the performance of tile defect classification, we compare performances of our model on dataset 1 and dataset 3.

Table 7. Performance comparison results of our model on dataset 1 and dataset 3

From the results presented in Table 7, the masks added to the defect images in dataset 1 significantly affect our CNN model's performance of ceramic tile defect classification. Without the masks, the defect detection accuracy decreases to 97.30%, with a precision of 97.56% and recall of 99.20%. However, with the addition of the masks of the defect images based on segmentation and morphological operations and HSV color mode, a performance improvement is achieved by 2.63% and 2.44% for accuracy and precision, respectively.

6.3 Quantitative comparison

In this section, we compare the performance of our model with existing state-of-the-art methods: VGG16-19 [28], AlexNet [29], and ResNet50 [30]. The training of these models on the dataset takes place with the same hyper-parameters mentioned in the previous section. We used two scores: the overall accuracy of classification and the number of network parameters. The results obtained after training these models on our dataset composed of two classes with defects and without defects are presented in Table 8. We observe that our model, based on a convolutional architecture and integrating a CBAM attention module, outperforms the four models in terms of accuracy. Furthermore, our model produces good results with a significantly reduced number of parameters compared to existing models.

Table 8. Quantitative comparison between our model and the state of the art methods

6.4 Ablation study

We conducted an ablation study to show the effectiveness of the attention module in our architecture.

In this experiment, we investigated the CBAM attention module's effect on ceramic tiles' defect classification results. Table 9 compares our CNN architecture without CBAM attention module integration which noted CNN_base and our proposed model with CBAM attention module integration.

Table 9. Ablation study results of the CBAM module on the classification performance of our model

These results show that our model based on the CBAM module produces good results compared to the basic model in terms of accuracy, precision, and recall. Indeed, integrating the CBAM module provides more efficiency and precision to the tile classification results on our dataset. Furthermore, these results show that the CBAM attention module allows us to focus on the ceramic tiles' relevant features to localize the defects, which allowed us to increase the performance of our model.

The accurate detection of defects in ceramic tiles, until now, is a significant issue for researchers due to the difficulty of distinguishing between defects and surface texture. Therefore, in this paper, to address this challenge, we propose a solution for accurate defect detection of ceramic tiles based on deep learning and attention mechanism.

Our solution consists of three fundamental steps: the first step consists in collecting real images from a production unit, preprocessing, and classification.

Our model is capable of classifying with high precision the three defects (corner breaks, pinholes, and pattern discontinuity) through a binary classifier based on the convolutional architecture equipped with a CBAM attention module. Furthermore, given the complexity imposed by the defects of the ceramic tiles and to simplify the learning of the ceramic defects, a preprocessing step is carried out on the images of the data set by applying several masks according to each defect in order to expose the characteristics of defects better, this simplifies learning and improves the performance of classifying ceramic tile defects.

Integrating the CBAM module into our CNN model allows our model to extract the local and global characteristics of defects, which greatly improves the performance of our model and produces good results compared to state-of-the-art methods.

As future work, we want to integrate our model into an industrial vision system capable of providing all the essential information concerning the defects of the ceramic tiles to the control room. Indeed we plan to improve our model to offer a muti-class classification that includes more defects and exact identification of the ceramic tiles by using RFID tags attached to the ceramic tiles to send the classification result to the processing center via a Zigbee protocol.