Leveraging Fine-Grained Deep Convolutional Neural Network for Automated Wafer Map Defect Detection and Classification in Semiconductor Manufacturing

Currently, the semiconductor industry is developing quickly, and as a result of considerable technological developments, more classified products are being designed and produced [1]. Semiconductor devices are occupying almost every aspect of the modern world, from smartphones to ultra-high specification television sets [2]. To sustain the ever-increasing demand for low-priced, higher memory capacity devices, and/or faster computing power, the design of the semiconductor production process is developing into a more difficult process that needs the introduction of complex assemblies, new manufacturing methods, new hardware components, and strict monitoring of cleaning and managing the systems [3]. The introduction of innovative aspects of the production procedure could be a main source of unwanted particles of material deposition. Also, modifications in present process conditions can result in particle generation based on the reaction dynamics of the model [4]. These particle faults on semiconductor wafers could be one of the major reasons for product failure. To understand the effect of production tools on their faults, the standard practice in the semiconductor material industry is to process thousands of wafers per annum per classification tool, like optical scattering and scanning electron microscopy tools [5]. Every wafer can consist of tens of faults. As the production process and, therefore, particle defect configurations become more difficult, the quantity of defects considerably increases.

Wafer production involves many intricate phases and variables. In the modern era, the growing difficulty and density of wafer designs have caused an increase in defect variety inside semiconductor wafers [6]. The classification of these defects provides a particular difficulty in relating to the unique defects because of their variations in orientation and location in the wafer [7]. Therefore, the effective categorizations of such defects signify a crucial analysis endeavor in semiconductor production procedures. The Wafer map classification and Manual feature extraction pose a significant problem relating to lab and time for engineers [8]. Recently, Artificial Intelligence (AI), especially machine learning (ML) methods, has established its efficiency in exactly identifying defects by extracting different features, surface textures, and critical pattern data. Particularly, deep learning (DL) models for semiconductor defect detection have been made a considerable process [9]. Researchers discovered that the utilization of convolutional neural networks (CNNs) has led to recent advancements in the state-of-the-art image classification operation, and developed a traditional method for addressing any image classification challenges. CNN is the end-to-end technique and does not need any task-specific feature engineering [10]. This end-to-end technique is useful as we don’t need to improve the task-specific feature extractors, and the field-specific expert knowledge is unnecessary. Another point of image classification is the difficulty of image recovery [11]. The image recovery is a challenge of finding images comprising the same scene or objects, providing query images, and it was employed in security and monitoring, medical images, and several fields [12]. Usually, the image recovery needs feature extraction employing object shape and colours. As the deep CNN can learn strong features at every layer, such standard features are utilized as good descriptions for image recovery [13]. CNN has presented an excellent operation compared to other classification methods, provoking a comprehensive study in the domain of wafer map pattern categorization.

To precisely identify wafer map defects in semiconductor manufacturing, this paper focuses on the design and development of a novel fine-grained deep convolutional neural network-based wafer map defect detection and classification (FGDCNN-WMDDC) approach. The proposed model intends to classify nine different types of wafer map defects using advanced models. This study follows a multi-faceted approach integrating preprocessing, feature representation, parameter tuning, and classification. The proposed model is evaluated on the benchmark datasets under several measures. The following are the major contributions of this research:

•Integrate Dual Attention EfficientNet architecture for robust feature extraction, which proficiently captures highly discriminative features contributing to the classification process.

•Utilize Nadam optimizer for hyperparameter optimization, which helps attain improved model convergence and better model performance with less computational complexity.

•Employ stacked sparse autoencoder (SSAE), a DL-based model for wafer defect detection and classification, are utilized and effectively classifies nine different classes.

•Conduct extensive experiments and demonstrate improved performance of the model compared to other baseline models across diverse measures.

This section defines the architecture and working principles of the proposed FGDCNN-WMDDC system for accurate wafer map defect detection and classification. Figure 1 depicts the overall structural design of the FGDCNN-WMDDC model. As depicted in Figure 1, images were first attained from WM811k Silicon Wafer Map Dataset Image dataset and then applied an adaptive bilateral filter (ABF) and contrast enhancement models to prepare the images for effective analysis by improving their quality. Following preprocessing, feature extraction takes place through a Dual Attention EfficientNet architecture. This model captures crucial features that influence the defect patterns. Then, the proposed model integrates the Nadam optimizer to fine-tune the model parameters for better performance. The selected feature sets are then input to the SSAE, which performs accurate defect type classification.

1.png

Figure 1. Overall structural design of the fine-grained deep convolutional neural network-based wafer map defect detection and classification (FGDCNN-WMDDC) model

3.1 Image preprocessing module

This subsection describes the preprocessing techniques used to improve the quality of wafer map images before feature extraction. In this study, an ABF and CLAHE are employed.

ABF: This method is broadly employed in image processing and CV for the edge-preserving smoothing process [22]. Unlike linear convolution filters, the BF employs a variety of kernels as well as a spatial kernel; these are commonly Gaussian. This module, which evades pixel blending with larger intensity changes, assures the protection of sharp edges, making the filter both non-linear and computationally in-depth. In ABF, the measure of the width and center of the range kernel of the Gaussian are permitted for modifying pixelwise.

Here, $f: \mathbb{I} \rightarrow \mathbb{R}$ denotes input imaging, $\mathbb{I} \subset \mathbb{Z}^2$ signifies the omain of image. $g: \mathbb{I} \rightarrow \mathbb{R}$ implies the output image.

$g(i)=\eta(i)^{-1} \sum_{j=\Omega} \omega(j) \phi(f(i-j)-\theta(i)) f(i-j)$         (1)

$\eta(i)=\sum_{j \in \Omega} \omega(j) \phi_i(f(i-j)-\theta(i))$         (2)

where, $\Omega$ signifies window centered with the origin, and $\phi_i: \mathbb{R} \rightarrow \mathbb{R}$ denotes local Gaussian range kernel:

$\phi_i(t)=\exp \left(-\frac{t^2}{2 \sigma(i)^2}\right)$         (3)

Notably, the center $\theta(i)$ in Eq. (1) and width $\sigma(i)$ in Eq. (3) were structurally differing operations. The spatial kernel $\omega: \Omega \rightarrow \mathbb{R}$ in Eq. (1) was a Gaussian equation as given by,

$\omega(j)=\exp \left(-\frac{\|j\|^2}{2 \rho^2}\right)$         (4)

CLAHE: CLAHE is employed to enhance contrast in wafer map imagery while boosting image quality and improving the performance of classification and detection for various defect forms. Initially, the wafer map imaging is separated into minor parts by nominal separation, and then equalization of the histogram is used for every separation. To equalize the value of gray in the images, hidden features of the image shape are more noticeable. The sharp corners of the imaging are prevented by limitations in the grey spectrum and area.

Where CLAHE is employed for limiting the amplification that is measured utilizing a clipping histogram at some level. The Rayleigh distribution is applied for clipping the histogram that is mathematically computed in Eq. (5).

$g=g_{\min }\left[2\left(\beta^2\right) \ln +\left(\frac{1}{1-p(F)}\right)\right]^{0.5}$         (5)

Here, $g$ and $g_{\min}$ denotes the determined and minimal value of pixel, $\beta$ denotes the threshold parameter, and $p(F)$ implies a distribution function. These improve wafer imaging support to the next stage of feature extraction.

3.2 Feature learning

This subsection specifies the extraction of meaningful attributes from wafer map images to assist in precise defect classification. In this paper, a Dual Attention EfficientNet model is utilized to extract both spatial and channel-wise relationships in wafer map defect patterns by a two-attention mechanism, which results in classifying changes among the wafer map defect classes. The primary goal of the attention mechanism is to choose a significant region of input information that doesn’t consider the areas with weak and non-discriminative defect attributes. These mechanisms concentrate on differentiating the feature extraction from several areas of wafer map imaging. In this study, the Dual Attention module is used to eliminate the interference of irrelevant data and focus more on the valuable defect-related data. Position Attention Mechanism (PAM) and Channel Attention Mechanism (CAM) have been integrated with EfficientNet-B0 to regulate the relations among the global and local features.

PAM: To extract the contextual information from lower‐level features, PAM is used for learning more information to enhance the feature representation. The PAM architecture is intended to gather and detect relevant features associated with the spatial domain. The major benefit of PAM is learning the spatial interdependency of features and gaining several contextual data points from local components. The input feature mapping $A$ is computed as in Eq. (6).

$A=F \in R^{C \times H \times W}$         (6)

Let, $F$ implies features; $C$ signifies the number of relevant features in the spatial domain, $H$ indicates the height of the spatial dimension; $W$ implies the spatial dimension width of the input tensor. The PAM makes a positional relations method between features for acquiring global feature data and integrates features randomly by the total weights allocated for features at every position. The input feature mapping performs a convolution function with a BN layer and a Rectified Linear Unit (ReLU) layer for providing the three novel feature maps F1, F2, and F3. These maps are known as single‐channel feature maps that are attained from F, the feature maps F1 and F2 are of similar dimensions, but F3 has various dimensions. After the performance of the reshape function on F1, F2, and F3, the scaling of feature maps becomes $H \times W$ then perform a matrix multiplication. Therefore, the last output $F \in R^{C \times H \times W}$ is developed over the multiplication of differentiated outcomes achieved by learnable parameters $\delta$ and an element-wise addition with the original feature mapping is expressed in Eq. (7).

$F^{\prime \prime}=\delta \cdot {reshape}\left(V \cdot M_p\right)+F^{\prime}$         (7)

CAM: Channel attention is used to update features of various channels to simulate the significance of each channel from the feature extraction. The CAM detects relationships in the local cross‐channel to evaluate the channel and its neighbours. Therefore, the number of parameters is lessened, and the model difficulty is also decreased without compromising the precision. In CAM, transition and dense block layers are combined to avoid adding many parameters that result in overfitting. The transition layer in CAM contains $1 \times 1$ a convolutional (Conv) layer as well as an average pooling layer with a stride of two for reducing the feature map size. This mechanism is divided into two stages, like the squeeze and excitation.

In squeeze stages, input attributes are compressed into one‐dimensional vector lengths that signify the number of channels. In the excitation phase, dependencies between the channels are gathered by a gate mechanism, which comprises dual non-linear fully connected (FC) layers. To squeeze and expand feature channels, CAM allocates adaptive weights to several features. The mathematical expression of the output of CAM is signified in Eq. (8).

$\beta(U)=\sigma\left(W_1 W_2\left(U_{a v g}+U_{\max }\right)\right)$         (8)

Here, $\sigma$ denotes the activation function of the sigmoid $W_1$ and $W_2$ implies the weights of the FC layer, $U_{a v g}$ and $U_{\max }$ denotes the average and maximal pooling feature map.

EfficientNet-B0: EfficientNet-B0 is a CNN technique that uses a complex scale approach for image recognition and identification. The network is scaling along with optimizing the width, depth, and resolution, which leads to enhancing the network. The primary goal of EfficientNet-B0 structure is to find an appropriate model for scaling the CNN technique to attain high precision for the recognition, classification, or detection process. Therefore, the scaled width of the network layers is obtained, resulting in more fine-tuned features and training the model. Nevertheless, wide and shallow layers in the network are not capable of learning from higher-level features, and high‐resolution imaging is employed for learning fine-grained and complex patterns. Later, regulating these depths, width, and resolution, EfficientNet-B0 enhances the classification execution.

Numerous Conv layers with 3 × 3 receptive fields and mobile inverted bottleneck Conv layers effectively attain information from the features extracted for wafer map defect classification. The scaling depth, width, and resolution in EfficientNet-B0 are shown in Eqs. (9)-(13).

$d=\alpha \varphi$         (9)

$w=\beta \varphi$         (10)

$r=\gamma \varphi$         (11)

$\operatorname{s.t} \alpha \cdot \beta 2 . \gamma 2 \approx 2$         (12)

$\alpha \geq 1, \quad \beta \geq 1, \quad \gamma \geq 1$         (13)

Here, $r$, $w$, and $d$ denote the network’s resolutions, width, and depth; constant terms $\alpha, \beta$, and $\gamma$ are attained employing the grid search parameter-graining approach. A coefficient is a user-defined variable that allocates scaled resources for each technique in the network. By fine-graining the resolution, width, and depth of the network, optimize the network precision and memory consumption consistent with the accessible sources. Unlike deep CNNs, a predefined set of scaling coefficients is employed in EfficientNet-B0 to regulate every dimension of layers in the network, which outperforms the prior cutting-edge techniques trained on ImageNet database. An incorporation of Dual Attention in Efficient-B0 improves the learning of the most important data about defect levels of the wafer map from pre-processed images. Figure 2 represents the model diagram of Dual Attention EfficientNet.

2.png

Figure 2. Model diagram of Dual Attention EfficientNet

3.3 Model optimization

This subsection presents the optimization strategy exploited to improve model performance. Nadam is an optimizer algorithm that integrates the adaptive learning rate mechanism of Adam with the lookahead gradient method of Nesterov accelerated gradient (NAG). Although the conventional momentum approach adjusts parameter upgrades that depend on collected gradients from prior phases, Nesterov momentum expects future gradient directions, allowing more informed upgrades. Nadam combines this foresight into Adam’s adaptive upgrade structure, enabling parameters for converging more effectively toward optimal solutions. This incorporation results in quicker convergence and enhanced stability in training, particularly in non-convex optimizer issues or when handling the noisy gradients. Therefore, Nadam is broadly accepted as an improved optimization in NN training, and it played an important part in enhancing training efficacy and overall model execution. The weights are upgraded based on:

$w_t^i=w_{t-1}^i-\frac{\eta}{\sqrt{v_t+e}} \cdot \widetilde{m}_t$         (14)   

$\tilde{m}_t=\beta_1^{t+1} \widehat{m}_t+\left(1-\beta_1^t\right) \hat{g}_t$         (15)

$\hat{m}_t=\frac{m_t}{1-\Pi_{i=1}^t \beta_1^i}$         (16)

$\hat{g}_t=\frac{g_t}{1-\Pi_{i=1}^t \beta_1^i}$         (17)

Here, $\eta$ denote the learning rate of the parameter, and $w_t$ implies weights at step $t$. Even though, $\beta_i$ signifies applied to choose the number of data required from the last upgrade, here $\beta_i \in[0,1], m_t$ denote the initial moment.

Table 1 summarizes the optimal hyperparameter configuration adopted to ensure stable and efficient model training. The EfficientNet backbone integrated with PAM/CAM is selected to enhance feature representation, while a high-resolution input size of 1024 × 1024 × 3 preserves fine-grained spatial details. The Nadam optimizer with a learning rate of 0.001 and momentum parameters (β₁ = 0.9, β₂ = 0.999) facilitates faster convergence and improved generalization. Additionally, a batch size of 32, 25 training epochs, ReLU activation, and a dropout rate of 0.5 collectively balance learning efficiency and overfitting control.

Table 1. Optimal value of the hyperparameter

3.4 Defect identification and classification module

This subsection describes the SSAE method utilized to classify different wafer defect patterns.

AE: AE has an unsupervised architecture as 3 layers NNs, involving a hidden layer (HL), input, and output layers. The AE could progressively alter a certain feature vector into an abstract feature vector, which could well understand the non-linear transformation from higher-dimension to lower-dimension data.

The procedure of encoding from the input layer to HL:

$H=g_{\theta_1}(X)=\sigma\left(W_{i j} X+\varphi_1\right)$         (18) 

The method of decoding from HL to the output layer:

$Y=g_{\theta_2}(H)=\sigma\left(W_{j k} H+\varphi_2\right)$          (19) 

Now, $X=\left(x_1, x_2, \ldots, x_n\right)$ denote the input vector of data, $Y=\left(y_1, y_2, \ldots, y_n\right)$ denote the reconstructed vector of input and $H=\left(h_1, h_2, \ldots, h_m\right)$ denote the outcome vector of lower dimension from HL. $W_{i j} \in R^{m \times n}$ denotes the weighted connection matrix among the input layer as well as HL. $W_{j k} \in R^{n \times m}$ implies the weight connection matrix among HL as well as the output layer. Here, $\varphi_1 \in R^{n \times 1}$ denote the biased vectors of the input layer and $\varphi_2 \in R^{m \times 1}$ signifies the bias vector of HL, correspondingly. $g_{\theta_1}$ signifies an activation function of HL neurons, and $g_{\theta_2}$ denote activation function output layer neurons, correspondingly, whose parts are mapping the outcome of the network to zero and one.

$g_{\theta_1}(\cdot)=g_{\theta_2}(\cdot)=\frac{1}{1+e^{-x}}$          (20) 

To adjust the parameters of the encoder-decoder, the error between the outcome reconstruction data and actual data could be lessened, and AE reconstructed the actual data in training. The function of reconstructive error $J_E(W, \varphi)$ among $H$ and $Y$ utilizes the mean squared error (MSE) operation; here $N$ denotes the number of input instances.

$J_E(W, \varphi)=\frac{1}{2 N} \sum_{r=1}^N\left\|Y^{(r)}-X^{(r)}\right\|^2$          (21) 

SAE: The objective of sparse coding is to computationally simulate the learnable receptive regions. For instance, the input data has been conveyed to the layer by a simpler copy. Even though the actual input data could be retrieved completely, the AE doesn’t extract any significant attributes in these instances. Consequently, the objective of the sparse coder proposes a sparse penalty in the HL of AE, thus the AE must produce shorter and more effective lower‐dimension output features with sparse restraints to enhance the expression of the input data. Let consider $\hat{\rho}_j, \hat{\rho}_j=\frac{1}{N} \sum_{i=1}^N\left[n_j\left(x_i\right)\right]$ represents the average activation of neurons in HL. The average activation $\widehat{\rho}$ methods a constant $\widehat{\rho}_j$ That is close to 0 values.

So, the Kullback-Leibler (KL) divergence is included as a standardization time to the error function of the AE to accomplish measures:

$K L\left(\rho \| \hat{\rho}_j\right)=\rho \log \frac{\rho}{\hat{\rho}_j}+(1-\rho) \log \frac{1-\rho}{1-\hat{\rho}_j}$          (22) 

The error function of sparse AE contains dual classes: the initial term was an MSE term, and the next one arises at the time of regularization.

$J_{{sparse }}(W, b)=J(W, b)+\mu \sum_{j=1}^m K L\left(\rho \| \hat{\rho}_j\right)$          (23) 

Now, $m$ denotes the amount of HL and $\mu$ imply weight factor that regulates sparse item strength. Moreover, prevents overfitting and adds the weighted attenuation parameter for the error function, $\lambda$ implies the coefficient of weight.

$\begin{array}{r}J_{{sparse }}(W, b)=J_E(W, b)+\mu \sum_{j=1}^m K L\left(\rho \| \hat{\rho}_j\right) +\frac{\lambda}{2} \sum_{r=1}^3 \sum_{i=1}^m \sum_{i=1}^{m+1}\left(w_{i j}^r\right)^2\end{array}$          (24) 

SSAE: This method is an NN that contains multiple SAE linked end-to-end. The outcome of the prior layer of sparse self-encoders has been employed as the input of the following layer of self-encoders, and then the high-ranking feature representations are achieved.

The greedy layer-wise pre-training approach has been employed for sequential training of every layer of SSAE to gain admittance to the optimized connection weighted and biased values of the total SSAE network. Therefore, the error back propagation approach has been leveraged to modify the SSAE while waiting for the outcome of the error function among input and output features to satisfy anticipated needs.

For the error function $J_{{sparse }}(W, b)$ described.

$\begin{aligned} & \frac{\partial}{\partial w_{i j}^J} J_{ {sparse }}(W, b)=  \frac{1}{2 n_r} \sum_{r=1}^{n_r} \frac{\partial}{\partial w_{i j}^J} J_{ {sparse }}(W, b, X(n), Y(n))+\lambda w_{i j}^r\end{aligned}$          (25) 

$\begin{aligned} & \frac{\partial}{\partial b^r} J_{{sparse }}(W, b)=\frac{1}{2 n_r} \sum_{r=1}^{n_r} \frac{\partial}{\partial b^r} J_{{sparse }}(W, b, X(n), Y(n))\end{aligned}$          (26) 

Hence, the upgrade procedure of weight and bias is as follows:

$w_{i j}^k=w_{i j}^k-\eta \frac{\partial}{\partial w_{i j}^k} J(W, b)$          (27) 

$b^r=b^r-\eta \frac{\partial}{\partial b^r} J(W, b)$          (28) 

Let $X(n)$ and $Y(n)$ denotes $n_{th}$ actual and reconstructed vectors; $\eta$ implies an upgraded learning rate.

In this manuscript, the FGDCNN-WMDDC model has been presented for detecting and classifying diverse kinds of manufacturing defects in silicon wafers, namely center, donut, edge local, edge ring, local, near full, none, random, and scratch. The FGDCNN-WMDDC approach follows an end-to-end architecture encompassing pre-processing, feature representation, and classification. Primarily, the proposed system applied ABF and contrast enhancement procedure as a pre-processing step. Following that, the Dual Attention EfficientNet architecture is implemented, which can dynamically learn and capture robust classification features. To enhance the performance of the DL method, the hyperparameter selection is carried out through Nadam optimizer. Lastly, SSAE is integrated for the identification and classification of different defect types. The performance validation of the proposed FGDCNN-WMDDC method is examined on WM811k Silicon Wafer Map Dataset Image dataset. Comprehensive experimental results underscored that the proposed method performed better compared to other techniques concerning multiple metrics. Thus, the proposed method can be implemented as an effective tool for automated detection and classification of distinct defects that exist in the wafer map.