Enhanced Material Classification via MobileSEMNet: Leveraging MobileNetV2 for SEM Image Analysis

The burgeoning field of nanotechnology is rapidly expanding on a global scale, significantly influencing advancements in a multitude of disciplines [1]. Nanoscience explores phenomena and material behaviors that emerge distinctly at the atomic, molecular, and macromolecular scales, where materials exhibit unique properties. It is widely acknowledged that nanotechnology is central to the technological advancement and societal development of nations [2]. This significance is underpinned by the unique characteristics of matter and energy manifest at the nanoscale. As particles are reduced from macroscopic to nanoscale dimensions, their physical and chemical interactions are altered, often enhancing quantum mechanical properties. This transformation in geometric configurations can lead to substantial changes in material structure. Nanotechnology, therefore, can be defined as the precision engineering conducted within the scale of 0.1 nm to 100 nm [3], offering the capability for the versatile manipulation of materials.

Scanning Electron Microscopes (SEMs) utilize focused electron beams to scan the surfaces of samples [4]. In SEM imaging, the generation of images is achieved by directing high-energy electrons toward the sample, detecting the resultant interactions between the electron beam and the sample atoms, and amplifying these signals for visualization on a cathode ray tube. These high-resolution images reveal the intricate details at nanometer scales, aiding the analysis of scalable surface structures. However, the manual interpretation of SEM images is inherently subjective and labor-intensive. The integration of machine learning models presents a promising avenue for the automation of SEM image classification, substantially enhancing the efficiency and objectivity of the analysis. Deep learning, with its capacity to analyze visual data through multi-layered neural networks, represents a paradigm shift in the classification of complex, high-dimensional SEM image datasets, which are pivotal in fields such as materials science, biology, and nanotechnology. The algorithms within these architectures can automatically discern and learn features inherent to the images for classification purposes. This article endeavors to contribute to this burgeoning field by examining various deep learning architectures for SEM image classification and elucidating their application in domains such as material characterization, biomedical research, and material design. This study aspires to be an instrumental guide for the more accurate and expedient classification of SEM images, thus bolstering their application in scientific and industrial research avenues.

Extensive studies utilizing deep learning and machine learning techniques have been conducted across various fields [5-9]. Dahy et al. [10] classified SEM images of 750 nano-scaled materials using a VGG-19 network augmented with Gray Wolf Optimization, achieving an accuracy of 97.00% and an F1-score of 98.00%. Modarres et al. [11] explored the automatic classification of SEM images through transfer learning, manually categorizing 20,000 images into 10 classes and attaining an accuracy close to 90% on their test dataset. However, they noted a decrease in accuracy with images that featured elements from multiple categories. Aversa et al. [12] applied transfer learning with CNN architectures and adapted ImageNet features to an SEM dataset with fine-tuning, achieving a 73.00% accuracy using AlexNet across 7557 images in 10 classes. Chou et al. [13] combined a decision tree with a neural network to classify defects in chip-scale package images, employing a preprocessing step to quantify various wafer surface features before classification. Dey et al. [14] leveraged VGGNet architectures to classify fractures and line collapses in SEM images. Imoto et al. [15], focusing on semiconductor manufacturing errors, utilized a 33-layer CNN with transfer learning on a dataset of 5388 images across 12 classes, achieving a precision and recall of 88.40%. Cheon et al. [16] reported a 96.2% accuracy in classifying wafer surface defects with a bespoke CNN, despite misclassifying 12 out of 319 defective test images. Barua et al. [17] achieved 99.53% accuracy for four classes and 97.09% for 10 classes using DenseNet201. Lastly, Chang et al. [18] employed a hybrid approach to classify defects, integrating morphological processes and thresholding techniques for high-density defects and developing a simple boundary defect detection algorithm for low-density defects, resulting in 92.80% training accuracy and 86.00% test accuracy.

These literature examples highlight the diversity of deep learning approaches and their transferability to SEM image datasets. However, the optimal transfer of features from general datasets like ImageNet to specialized SEM images remains a challenge, warranting critical analysis to address potential domain gaps. Additionally, discussions on the effect of class imbalances on model performance and strategies to mitigate these effects are conspicuously sparse and warrant further exploration.

1.1 Literature gaps

The identified gaps in the literature, based on the preceding literature review, are articulated as follows:

The majority of machine learning (ML) models surveyed have been applied to datasets with a limited number of classes. The significance of this gap rests in the models' restricted capability to address real-world scenarios that encompass a much broader array of materials. This limitation curtails the scope and practical application of these models, highlighting the need for research into ML models that can handle more complex and varied class scenarios.

The classification performance of the models in question is relatively modest. This gap is of critical importance, as it directly affects the reliability and efficacy of ML models in practical applications. Exploring the underlying causes of this subdued performance and proposing actionable solutions are imperative steps towards bolstering model accuracy. Addressing this gap is essential to augment the credibility of ML models in material classification, thereby encouraging their broader integration across diverse industrial sectors.

To the best of our knowledge, there is an absence of fixed-size patch-based deep learning models tailored to SEM image classification. This gap highlights an innovative opportunity in model architecture that is specifically designed to cater to the distinct characteristics of SEM images. The development of a fixed-size patch-based deep learning model could potentially provide a novel approach to feature extraction and classification accuracy, revolutionizing SEM image classification. Such a model could offer a more potent and flexible solution to the challenges presented by varying image sizes and complexities.

By addressing these gaps, researchers can substantially propel the field of SEM image classification forward. Such advancements would likely result in more sophisticated and versatile ML models, heightened classification performance, and the creation of specialized architectures that are finely attuned to the specificities of SEM images. These improvements would collectively enhance the practicality and impact of these models in the realm of material science and beyond.

1.2 Motivation and our model

Material science stands as a discipline with expansive applications, and an array of models have been proposed by researchers to tackle its inherent challenges. With machine learning models demonstrating notable adeptness in managing non-deterministic problems, their integration into various research fields has surged. The impetus of this study is to unveil a novel machine learning model specifically tailored for the classification of Scanning Electron Microscope (SEM) images. Our approach harnesses the capabilities of the mobile deep learning architecture, MobileNetV2, to train our dataset.

Introducing MobileSEMNet, our model emerges as a deep feature engineering innovation. It is predicated on a patch-based architecture deploying fixed-size patches, a concept inspired by the success of transformers like the Swin Transformer and Vision Transformer, which have shown remarkable prowess in classification through patch utilization.

MobileSEMNet operates by engaging fixed-size patches to meticulously extract features from SEM images. In this dual-input feature extraction process, patches and the overall image serve as the foundational inputs. Deep features are synthesized through the Global Average Pooling (GAP) layer intrinsic to MobileNetV2 and are subsequently combined to craft the final feature vector. The feature selection phase sees the application of Neighborhood Component Analysis (NCA) to distill the most informative features, which then serve as the input for the Support Vector Machine (SVM) classifier.

The rationale behind selecting MobileNetV2 as the cornerstone for MobileSEMNet is anchored in its demonstrated efficiency and classification prowess within the realm of image classification, particularly in scenarios bounded by computational restrictions. With an architecture celebrated for its lightweight design-primarily due to depthwise separable convolutions-MobileNetV2 balances computational frugality with robust classification capabilities, rendering it an exemplary fit for processing voluminous SEM image datasets.

The patch-based methodology integral to MobileSEMNet is an innovative leap from conventional, holistic image processing techniques. This method excels in delineating localized patterns and variances within SEM images, culminating in enhanced precision in material classification tasks. The fusion of deep feature engineering and SVM classification further augments the model's facility to adeptly navigate the multifaceted landscape of material science challenges.

In essence, our MobileSEMNet is a distinct and pioneering model that amalgamates the efficiency of MobileNetV2 with a sophisticated patch-based approach. By leveraging cutting-edge deep learning methodologies, MobileSEMNet is adeptly positioned to fulfill the specialized requirements of SEM image classification within the domain of material science.

1.3 Contributions

This research has yielded several significant contributions:

• We have pioneered a novel deep feature engineering model, which we have termed MobileSEMNet. This model has been conceived with the explicit intent of bridging the literature gaps identified in SEM image classification. It offers an advanced methodology for feature extraction and representation, tailored for the unique characteristics of SEM imagery.
• MobileSEMNet has undergone extensive evaluation on a sizable dataset of SEM images, where it has attained an exemplary classification accuracy rate of 96.87%. This marked accomplishment underlines the effectiveness of our proposed model in bolstering the accuracy of SEM image classification. The high precision achieved by MobileSEMNet is a substantial stride forward, addressing the previously noted gap concerning the suboptimal classification performance of extant models. The practical import of this achievement is considerable: MobileSEMNet has been validated as a highly accurate tool for the categorization of materials based on SEM images. This contributes to more reliable and credible material analyses within the scope of material science, supporting the domain's progress through improved analytical precision.

The recommended MobileSEMNet is a MobileNetV2-based deep feature engineering model. In the first phase of our model, we have trained the training dataset on the MobileNetV2 [20] and we have obtained a pretrained dataset. To extract deep features, GAP layer of the used MobileNetV2 have been used. Fixed size patches have been used. 16 fixed size patches have been created for this model. We have extracted deep features from each patch and the whole SEM image. We have used the whole SEM image to extract global features. The fixed size patches have been used to generate local deep features. The generated 17 feature vectors have been concatenated to create the final feature vector. In the feature selection phase, we aimed to choose the most informative features from the created final feature vector. Therefore, we have used NCA feature selector and the most informative 512 out of the generated 21,760 features (=17×1280). In the last phase, we have classified the selected features using the SVM classifier with 10-fold cross-validation (CV).

The schematic explanation of the proposed MobileSEMNet is demonstrated in Figure 1.

Herein, p is fixed size patch with a size of 56×56 and f is the feature vector with a length of 1280.

To better explain the proposed MobileSEMNet, the steps of this model are:

Step 1: Resize each image to 224×224 sized images.

Step 2: Train the used training dataset using MobileNetV2 and obtain a pretrained CNN.

Step 3: Divide test images into 56×56 sized patches. In this step, 16 patches have been created.

Step 4: Extract deep feature from each test image and patches of them by deploying GAP layer of the pretrained MobilNetV2 and create feature vectors.

$f_1=M(I m, G A P)$      (1)

$\begin{gathered}p_k=\operatorname{Im}(56 \times(i-1)+1: 56 \times i, 56 \times(j-1)+1: 56 \times j,:), \\ j \in\{1,2,3,4\}, i \in\{1,2,3,4\}, k \in\{1,2, \ldots, 16\}\end{gathered}$   (2)

1.png

Figure 1. Graphical explanation of the proposed MobileSEMNet

$f_{k+1}=M\left(p_k, G A P\right)$     (3)

Herein, $f$  is the generated feature vector by the GAP layer of the pretrained MobileNetV2 ($M(.,$.)). In Eq. (2), we have mathematically defined the fixed size patch division and $p$ implies the patch with a size of 56×56. By using the GAP layer of the pretrained MobileNetV2, 1280 features have been extracted. Therefore, the length of feature vector is equal to 1280. In this step, we have created.

Step 5: Merge the extracted features to get final feature vector.

$X=\omega\left(f_1, f_2, \ldots, f_{17}\right)$       (4)

Herein, $X$ defines the merged feature vector and we have created the merged feature with a length of 21,760 (=1280×17).

Step 6: Apply NCA [21] to the generated features to select the most informative 512 feature. A brief information about the NCA feature selector has been given below.

NCA is the commonly used feature selector in machine learning and is a distance-based method. Its central aim lies in augmenting classifier performance by selecting the most informative features from the generated features.

The core concept driving NCA revolves around advancing classification accuracy by maximizing the relative similarity among data points belonging to the same class while minimizing the similarity between data points from distinct classes. In essence, NCA seeks a transformation that draws similar data points closer together, while pushing dissimilar data points further apart.

Using the optimization of a differentiable objective function, NCA acquires the transformation matrix that best distinguishes between classes. Characterized as a distance metric learning approach, NCA tries to learn a distance metric that reflects the intrinsic data structure, rendering it particularly suitable for the specific classification task at hand.

NCA has demonstrated success in diverse domains, encompassing computer vision, natural language processing, and bioinformatics. Its adaptability to the data and its proficiency in selecting informative features have rendered it a valuable instrument in bolstering classifier performance and elevating the overall accuracy of machine learning models.

Step 7: Classify the selected features by deploying the SVM [22, 23] classifier with 10-fold cross-validation.

The given seven steps above have been defined the proposed MobileSEMNet. Moreover, we have explained the methods as below.

We selected MobileNetV2 as the basis for MobileSEMNet due to its proven efficiency in image classification tasks, especially in resource-constrained environments. The model's lightweight architecture, utilizing depthwise separable convolutions, strikes a balance between computational efficiency and high-performance classification. This choice was motivated by the need for a model that could handle large SEM image datasets effectively while ensuring computational feasibility.

Global Average Pooling (GAP) layers were employed to extract deep features from SEM images. GAP layers condense spatial information into a single value per feature map, allowing the model to capture essential patterns and features efficiently. This is particularly advantageous in reducing the dimensionality of the feature space, promoting computational efficiency, and preventing overfitting. The choice of GAP layers aligns with the objective of MobileSEMNet to efficiently capture relevant features from SEM images.

The selection of a fixed-size patch with dimensions of 56×56 was deliberate. This choice aims to balance granularity and computational efficiency. A patch size of 56×56 provides sufficient local information for capturing fine-grained details in SEM images while keeping computational demands manageable. This decision was made to strike a pragmatic balance between feature richness and model efficiency.

In the feature selection phase, we opted to choose the most informative features using the NCA feature selector. The selection of precisely 512 features was driven by a desire to strike a balance between informativeness and computational efficiency. This number was chosen to ensure that the selected features encapsulate the most crucial information for accurate classification while avoiding excessive computational overhead.

In the classification phase, we have used SVM since we obtained the best classification performance with Cubic SVM. Moreover, we have used 10-fold cross-validation to get robust results.

Providing this reasoning aims to enhance the transparency of our methodology, showcasing the thoughtful considerations behind each decision. These choices collectively contribute to the effectiveness of MobileSEMNet in addressing the challenges posed by SEM image classification in material science applications.

3.png

Figure 3. The class-wise results

4.png

Figure 4. Performance comparison of the Chi2, ReliefF and NCA feature selectors

5.png

Figure 5. Accuracies of the used classifiers

This research introduces a novel deep feature engineering model named MobileSEMNet, implemented using MobileNetV2. The proposed MobileSEMNet achieved a commendable classification accuracy of 96.87% and an unweighted average recall (UAR) of 93.72% when evaluated on the utilized imbalanced SEM image dataset. Additionally, we present class-wise results, which are depicted in Figure 3.

According to Figure 3 and Table 2, the first class (Biological) achieved the highest precision and F1-score, while the fourth class (MEMS device and electrodes) attained the best recall. Conversely, the third class (Film-coated surface) displayed the weakest performance. Notably, all class-wise test results surpassed 80%, underscoring the MobileSEMNet's impressive capability in SEM image classification.

During the feature generation phase, we employed a patch-based deep feature extraction approach. By utilizing the Neighborhood Component Analysis (NCA) technique, we selected the top 512 features from the pool of 21,760 generated features. The outcomes of this process, specifically the selected features per the patches, are presented in Figure 4. We also conducted tests using other feature selectors, namely NCA, ReliefF, and Chi2. Figure 4 illustrates the classification performances of these selectors for our dataset.

As shown in Figure 4, it is evident that the NCA feature selector yields the best results among all the feature selectors considered. Hence, we have chosen to employ the NCA feature selector in the proposed MobileSEMNet.

For the classification phase, we evaluated the performance of six classifiers: (i) Decision Tree (DT), (ii) Linear Discriminant (LD), (iii) Naive Bayes (NB), (iv) k-Nearest Neighbors (kNN), (v) Support Vector Machine (SVM), (vi) Ensemble Discriminant (Ens. D.), and (vii) Neural Networks (NN). The classification performances of these classifiers have been depicted in Figure 5.

Based on the findings from Figure 5, the SVM classifier emerges as the most effective classifier when utilizing the generated features through MobileSEMNet. Consequently, we have selected SVM as our classifier of choice.

To demonstrate the effectiveness of MobileSEMNet, we conducted a comparison of classification performance between MobileNetV2 and our proposed model. It is important to note that we utilized the pretrained MobileNetV2 to create MobileSEMNet. As a result, we present the training and validation curves of MobileNetV2 for this dataset in Figure 6.

6.png

Figure 6. The training and validation curve of the MobileNetV2 for the collected dataset

7.png

Figure 7. Test accuracies of the MobileNetV2 and MobileSEMNet

As depicted in Figure 6, MobileNetV2 achieved a training accuracy of 100% and a final validation accuracy of 95.47%. Subsequently, we utilized this trained MobileNetV2 to construct the proposed MobileSEMNet. The main purpose of MobileSEMNet is to enhance the classification capability of MobileNetV2. The test accuracies of both models are illustrated in Figure 7.

As depicted in Figure 7, the proposed MobileSEMNet exhibits a notable increase of 2.02% in test accuracy compared to MobileNetV2.

Table 3. Comparisons

In order to showcase the high classification performance of MobileSEMNet, we conducted a comprehensive comparison with other state-of-the-art (SOTA) material classification methods. The comparative results have been presented in Table 3.

The comparative evaluation of various models in SEM image (see Table 3) classification reveals insightful findings. Wong et al. [29] implementation using GoogleNet for a dataset comprising 3 classes and 456 SEM images achieved an accuracy of 92.70%, demonstrating commendable performance. Azimi et al. [30] adopted a more complex approach with a Fully Convolutional Neural Network, DeCAF features, and SVM for a dataset containing 4 classes and 4331 patches. Despite the increased complexity, their accuracy reached 93.94%, showcasing the effectiveness of their methodology. In contrast, our proposed MobileSEMNet, employing MobileNetV2, NCA, and SVM for a more extensive dataset of 10 classes and 21,272 SEM images, achieved a notable accuracy of 96.87%, along with high recall, precision, and F1-score values (93.72%, 95.97%, and 94.79%, respectively). This indicates that our model outperforms the aforementioned studies, emphasizing its efficacy in handling a more diverse and extensive SEM dataset. The meticulous design and integration of deep learning features in MobileSEMNet contribute to its superior performance, showcasing its potential for advancing SEM image classification in material science and related domains.

The findings about the experiments of this research have been given below.

- In addition to overall accuracy, we employed several widely recognized performance evaluation metrics to provide a more nuanced understanding of the model's effectiveness. These metrics included: Recall: Measures the ability of the model to correctly identify instances of a particular class. Precision: Reflects the accuracy of the model when it predicts a specific class. F1-Score: Represents the harmonic mean of precision and recall, offering a balanced measure of a model's performance.

- Class-wise results were presented, depicting individual performance metrics for each class. This detailed breakdown highlighted the strengths and weaknesses of the model in classifying different categories. Visual representations, such as Figure 3, provided additional insights into the varying performance across different classes.

- UAR was employed as an additional metric, providing a balanced measure of recall across all classes. This metric is particularly useful when dealing with imbalanced datasets, ensuring that the model's performance is not skewed by the prevalence of any specific class.

- The performance of MobileSEMNet was compared with other state-of-the-art material classification methods, as demonstrated in Table 3. This comparative analysis provided context for evaluating the model's performance against existing benchmarks in the field.

- The training and validation curves of MobileNetV2 for the collected dataset were presented in Figure 6. This representation offered insights into the model's training progress, showcasing the convergence and potential signs of overfitting or underfitting.

- Test accuracies of both MobileNetV2 and MobileSEMNet were illustrated in Figure 7. The comparison highlighted the improvement achieved by MobileSEMNet over MobileNetV2, demonstrating the model's capacity to enhance classification capabilities.

- The selection of the Cubic SVM classifier was based on its superior performance during extensive comparisons. The hyperparameters were carefully chosen to optimize the model's accuracy while mitigating overfitting.

Advantages and limitations of this research are also given below.

Advantages:

- MobileSEMNet achieved a commendable classification accuracy of 96.87% on a diverse SEM image dataset, demonstrating its effectiveness in accurately identifying materials.

- The model provided detailed class-wise results, enabling a nuanced understanding of its performance across different material categories. This information is valuable for applications where specific classes hold more significance.

- The use of the Neighborhood Component Analysis (NCA) technique in feature selection enhanced the model's ability to focus on the most informative features, contributing to its overall classification performance.

- Leveraging MobileNetV2 as the base model ensured a balance between computational efficiency and classification accuracy, making MobileSEMNet suitable for real-world applications with resource constraints.

- The proposed MobileSEMNet demonstrated a notable increase of 2.02% in test accuracy compared to MobileNetV2, showcasing its capability to improve upon the baseline model.

- Patch-based Approach: The patch-based deep feature extraction approach allowed the model to capture fine-grained details in SEM images, enhancing its ability to recognize intricate patterns.

Limitations:

- The MobileSEMNet model is the manual selection of hyperparameters. While the chosen hyperparameters, including a 3rd-degree polynomial kernel, a box constraint level of 1, and one-vs-all coding, contributed to the model's superior performance, the manual selection process introduces subjectivity. Automated hyperparameter optimization techniques, such as grid search or random search, were not employed in this study. The use of automated methods could potentially fine-tune the hyperparameters more objectively, ensuring optimal performance and generalizability across different datasets.

- The dataset used for training and evaluation may exhibit class imbalances, potentially leading to biased results and affecting the model's performance on minority classes.