Transfer Learning Technique with EfficientNet for Facial Expression Recognition System

Several techniques have recently been used to investigate the facial expression recognition system. Algorithm convolutional neural networks (CNN) as a feature extraction process from the image then enter the classifier process using a neural network. Therefore, recent research using the deep learning method for the task of recognizing facial expressions by using a combination of the two processes above. Several studies have been reviewed and compared with existing FER system methods [10-13], From those based on Deep learning methods among them [12, 14]. The following is a brief explanation of some of the well-known techniques used in the method of making the Facial Expression Recognition (FER) system.

2.1 Machine learning-based FER approach

Automatically making a Facial Expression Recognition (FER) system is a challenge in research, especially in machine learning. Several traditional machine learning algorithms, such as K-nearest neighbor and neural networks, have been used to test making the FER model.

Xu and Wei [14] pioneered the FER method, which first added a wavelet energy feature (WEF) to the face image, then used the Fisher's linear discriminants (FLD) method for feature extraction, then continued with the expression classification process using the K-nearest neighbor (KNN) method. The KNN method was also used by Zhao et al. [15] in order to categorize facial expression recognition. However, they employed non-negative matrix factorization (NMF) and principal component analysis (PCA) for the feature extraction procedure.

Feng et al. [16] employed linear programming (LP) methods to categorize face expressions, local binary pattern (LBP) histogram extraction from various little portions of the image, and ultimately, feature histogram combination. Zhi and Ruan [17] facial feature vectors derived from 2D discriminant locality preservation projections. Lee et al. [18] employed a boosting method to classify the data and extend the wavelet transform to 2D, which it is known as the contourlet transform (CT), for the feature extraction procedure from the picture. Changand Huang [19] integrated face recognition into the FER system to improve each person's skill at expression recognition. For classification, they used a neural network and a radial basis function (RBF).

One of the methods used for classifying facial expressions is the Support vector machine (SVM) from images that have undergone a feature extraction process using distinct techniques. In this category, Shih et al. [20] investigate several feature representations (such DWT and PCA), and it has been found that DWT combined with 2D-linear discriminant analysis (LDA) performs better than other methods.

In this comprehensive investigation, they assessed various facial representations based on local statistical characteristics and LBP with various SVM variations [21]. Jabid et al. [22] investigated the local directional pattern (LDP), an appearance-based feature extraction method. Alshami et al. [23] SVM method was used to study two feature descriptors, facial landmarks descriptor and center of gravity descriptor. SVM and numerous other methods (e.g., KNN, LDA) were taken into account in the comparative analysis conducted by Liew and Yairi [11] for the classification of features extracted using various methods, including Gabor, Haar, and LBP. A recent investigation by Joseph and Geetha [24] based on their proposed facial geometry investigated various classification algorithms, including logistic regression LDA, KNN, classification and regression trees, naive Bayes, and SVM. The main limitation of the conventional methods mentioned above is that they only consider the frontal view for FER due to its features.

2.2 Deep learning-based FER approach

Creating facial expression recognition systems with deep learning approaches is relatively new in the machine learning [25] field. Until now, research on Convolutional Neural Networks (CNN) has been published in specialized journals. Zhao and Zhang used a deep belief network (DBN) algorithm and Neural Network for FER, where DBN was used for the unsupervised feature learning process. Then a neural network was used for facial expression classification. Pranav et al. [26] investigated FER on self-collected facial expressional images using a standard CNN architecture with two convolutional-pooling layers.

A larger architecture with four inception layers and two convolutional-pooling layers was investigated [27]. Pons and Masip [28] use a 72-CNN ensemble, with each CNN trained with a different filter size in the convolution layer or a different number of perceptions in each fully connected layer. Wen et al. [29] also use CNN ensembles, but they train 100 CNN and only use a subset of them in the final model. Ruiz-Garcia et al. [30] CNN sevens are trained with face images and initialized with encoder seven of the convoluted auto-encoder. This type of CNN initialization outperformed CNN with random initialization. FaceNet2ExpNet is a CNN architecture that extends the facial recognition architecture into FER [31]. Furthermore, Li et al. used transfer learning to create the FaceNet2ExpNet architecture. Jain et al. [32] consider the FER system's hybrid deep learning architecture, which includes the CNN algorithm and a recurrent neural network (RNN).

Shaees et al. used a hybrid architecture with a transfer learning approach [26], where SVM was used to identify the features of the AlexNet pre-trained architecture. Recently, CNN is used for the FER system, and DWT is used for feature extraction [33]. For the FER system, a relatively deep CNN architecture with 18 convolutional layers and four subsampling layers is used [34]. Recently, FER using a clustering approach with CNN [35]. Ngoc et al. [36] graph-based CNN was investigated for FER of natural facial features. Jin et al. [37] in their CNN-based method, they used both unlabeled and labeled data. In addition, Porcu et al. [38] used different data augmentation techniques, including synthetic images to train deep CNN architectures and combined synthetic images with other methods to perform the FER system better. Existing deep learning-based methods have also taken frontal images into account, and most studies have even excluded profile view images from the data set in experiments to simplify the task [11, 23, 30, 39].

This section investigates the efficiency of the proposed FER system using transfer learning on the CK+ and JAFFE datasets. First, a description of the CK+ datasets and then the experimental setup is presented. Finally, the results of the proposed model on the CK+ datasets were compared with several existing methods to verify the effectiveness of the proposed FER model.  

JAFFE datasets are processed in the same manner. The section investigates the efficacy of the proposed FER system on CK+ datasets using transfer learning. Following a description of the CK+ datasets, the experimental settings, such as image preprocessing techniques with image augmentation, are presented. Finally, the proposed model's results on the CK+ datasets were compared to several existing methods to validate the proposed FER model's effectiveness [40].

4.1 Benchmark datasets

Several well-known public datasets are available for use in the FER system, such as (e.g., FER2013, JAFFE, CASIA-Web Face, and IMED). Specifically, in this paper, the researcher uses the CK+ (Extended Cohn-Kanade Dataset) datasets [42]. The CK+ datasets images are divided into eight expression classes: afraid, angry, disgusted, sad, happy, surprised, neutral, and contempt. The CK+ datasets are described briefly below, along with the reasons for their selection.

Figure 4 shows 327 image sequences with labeled facial expressions from the CK+ database. However, only the last frame of each image sequence is labeled with an expression. Consequently, to collect more images for training, in this study, we selected the last three frames of each sequence for training or validation. Next, the first frame of each of the 327 labeled sequences will be selected as the neutral expression. As a result, 1308 images with eight labeled facial expressions can be obtained from this dataset.

Figure 5 JAFFE (or JAFFE for short) dataset contains expression images of Japanese women from Kyushu University's Psychology Department. The JAFFE dataset was collected in a controlled environment to generate frontal facial expressions. Furthermore, JAFFE contains seven facial expression variants locally. The JAFFE dataset is small, with only 213 frontal images of 10 individuals; Figure 5 shows some examples. This dataset was selected to investigate how a small dataset responds to model training. Furthermore, many studies evaluate the FER model using the JAFFE dataset (e.g., [17, 21, 22]).

4.png

Figure 4. Sample images from CK+ dataset

5.png

Figure 5. Sample images from JAFFE dataset

In this paper, the researcher divides the CK+ and JAFFE datasets into two parts, respectively: 90% for training and 10% for testing. Each section is placed into two different folders. Training data is used in the training process for building a model. Testing data is used for the testing process after building the FER model.

4.2 Preprocessing

In this case, preprocessing is done so that the image can be studied optimally by the model. In this case, cropping the face is done to take only part of the face image, areas that are not faced are removed so that they are not too burdensome for computation, and the model does not learn information that is not important in the image [39, 41].

6.png

Figure 6. Image augmentation process

Deep learning requires many data variants to get optimal performance. Figure 6 data augmentation is a technique of manipulating an image without losing the essence or essence of the image. Augmentation performed Random Flip, Random Rotation, Random Height, Random Width.

4.3 Experimental setup

OpenCV library is used to do image cropping. When entering a model, the image will be resized to a size of 224 x 224 because, by default, the resolution size of the deep CNN architecture, especially EfficientNet, is 224 x 224. For the optimizer parameter, Adam is used by considering the learning rate value for the feature extraction process as 0.001 with 50 epochs. Then for the feature classification process, we unfreeze five to seven layers close to the output layer, then train them with a learning rate of 0.00001 and epochs from 51 to 100 epochs. The small learning rate value prevents us from destroying what EfficientNet has learned from ImageNet datasets. Meanwhile, only minor augmentation is applied to the data, with the augmentation settings Rotation: (−10° to 10°), Scaling factor: 1.1, and Horizontal Flip. Minor adjustments to the original image enhance its accuracy.

In this research, to train and experiment in terms of testing the performance of a state-of-the-art facial expression recognition model, the researcher used the TensorFlow framework, using the Python programming language with Keras. In this case, the researcher's training process used a GPU from google with specifications Tesla K80 Cuda Version: 11.2, NVIDIA- SMI 495.46.

How this add-in operates as follows First, when the model has gone through the fitting process. The next step of the model is saved into HDF5 format to ingest the weight from the results of the training that has been conducted by the model. Next, it uses the load_model() function to be accommodated in the model variables. Then the model is tested with 10% of the testing data using the model.predict() function. Testing data is data that is separate from training data. After testing the model, the calculation evaluation was conducted using performance metrics, classification reports and confusion matrix using the scikit learn library. Researchers tested each model individually.

4.4 Experimental results and analysis

In this experiment, the efficiency of the proposed model for processing benchmark datasets is CK+ and JAFFE. Because CNN are architectures that consist of the proposed building blocks. Initially conducted a series of experiments using the standard CNN model to identify the initial performance. Then look for the difference in the impact of fine-tuning using EfficientNet-B0. Finally, create a model with fine-tuning techniques using a pre-trained architecture from the EfficientNet family.

Table 1. Testing accuracies of standard CNN CK+ and JAFFE datasets for various image input sizes are shown

The test set accuracies for standard CNN with two layers with 5×5 size kernel, and 2×2 MaxPooling for various input sizes ranging from 360×360 to 48×48 are presented in Table 1 for both the CK+ and JAFFE datasets. The test size was chosen at random from 10% of the available data. The presented results are the best test set accuracies for a total of 50 iterations for a specific setting. For example, in Table 1, the results and analysis can be described using CK+ image input measuring 48×48, resulting in a testing accuracy of 95.45% because CK+ images have been preprocessed so that they have dimensions of 48×48 pixels before entering into a model and when inputting images measuring 360x360 produce accuracy 57.58% is because the image will be blurry because the sharpness is reduced due to the resizing process from 48×48 to 360×360. To use the JAFFE datasets in this observation, it was found that the best accuracy was obtained from the image sizes of 360×360 and 128×128, respectively 91.67% and 91.67%. This is because the sharper the image size, the information obtained from the model is smoother, and the sharper the model is smarter.

Because fine-tuning and its mode are critical in the proposed transfer learning-based system, experiments were conducted with various fine-tuning modes to gain a better understanding. On the CK+ and JAFFE datasets, Table 2 shows the accuracy of the testing data (10% of randomly selected data) of the proposed model with EfficientNet for different transfer learning modes.

In this experiment, the researcher compares two training modes and the features extraction model's histories with the fine-tuning model. In this case, the CK+ and JAFFE datasets have undergone the data augmentation process before entering each model. First, the researcher did it in EfficientNetB0 as the base model, then set up the base model and froze its layers (this will extract features). Then set up model architecture with trainable top layers. The tuning parameters given in this training are using the loss categorical cross-entropy function and the optimizer using the adam function with a learning rate value of 0.0001 each so that the model in updating the weights is softer and the model gets more information. A model with a lower learning rate (it's typically best practice to lower the learning rate when fine-tuning). Second, to apply fine-tuning of the model, the researcher performs Unfreeze all the layers in the base model, then proceeds to Refreeze every layer except the last six because this is for which the top six layers of EfficientNetB0 can be trained according to custom datasets about this research. Each training mode for each model uses 50 epochs for the feature extraction model and 50 epochs for the fine-tuning model. So, the model has already done 50 epochs (feature extraction). This is the total number of epochs we're after (50 + 50 = 100).

Table 2. Testing data accuracy with EfficientNetB0 from different training modes

To find a suitable and best model, researchers experimented with using the architecture, specifically the EfficientNet model family (EfficientNet-B0, EfficientNet-B01, Efficient-Net-B02, EfficientNet-B03, EfficientNet-B04, EfficientNet-B05, EfficientNet-B06, and EfficientNet-B07). The number with the model's name represents the EfficientNet B1 to B7 obtained using different scale coefficients in which different combined coefficient values produce EfficientNet B1-B7. Experiments were conducted on 10% of the data chosen at random as a test (i.e., 90% as a training set), and the accuracy of the test set after 50 iterations of fine-tuning is shown in Table 3. In the selected 10% test data case for the JAFFE data set, all models achieved 100% accuracy. The accuracy of the CK+ dataset, on the other hand, ranges from 95.45% to 99.57% at 10% of the selected test data. The CK+ data set is much larger than the JAFFE data set.

Table 3. Comparison of testing accuracies with various EfficientNet models on the CK+ and JAFFE

In Table 3, we can see that the model using EfficientNetB03 has an accuracy of 95.45%, especially studying the CK+ pattern. Exploring the JAFFE dataset of the EfficientNet architecture family starting from EfficientNetB1-B7 yielded 100.0% accuracy. Why this is different is because the CK+ datasets have more variety of images and a greater number of images which is 1308 out of eight classes of the sum of each expression. JAFFE only has 213 images and seven classes for each expression, so the EfficientNet model can be studied easily because it has little data variance. In studying patterns from the CK+ datasets, the EfficientNetB1-B7 model family has different accuracy because each EfficientNet category has several different parameter sizes ranging from the depth, width, and resolution scale of the network in making each CNN architecture to produce several models which are commonly called the EfficientNet model family.

To find a suitable and best model, researchers experimented using the architecture, specifically the EfficientNet model family (EfficientNet-B0, EfficientNet-B01, Efficient-Net-B02, Efficient-Net-B03, EfficientNet-B04, EfficientNet-B05, To EfficientNet-B06, and EfficientNet-B07). The number next to the model's name represents the EfficientNet B1 to B7 obtained by varying the scale coefficients, with different combined coefficient values producing EfficientNet B1-B7.

 Experiments were conducted on 10% of the data chosen at random as a test (i.e., 90% as a training set), and the accuracy of the test set after 50 iterations of fine-tuning is shown in Table 3. In the selected 10% test data case for the JAFFE data set, all models achieved 100 % accuracy. The accuracy of the CK+ dataset, on the other hand, ranges from 93.47% to 98.78% at 10% of the selected test data. This is due to the fact that the CK+ data set is much larger than the JAFFE data set.

In testing the performance of this testing, the researcher uses the F1-Score as Eq. (1). Performance metrics measurement because it is good for measuring imbalanced datasets in which the number of each Class is not balanced.

$F 1-$ Score $=\frac{2 * \text { precision } * \text { recall }}{\text { precision }+\text { recall }}$      (1)

Table 4 shows the classification of the training results conducted by the EfficientNetB0 model on 132 test images from the CK+ data set. in which the model predicts the image of sadness is misclassified as disgust, and surprise is misclassified as sadness.

Table 4. The F1-score classification of each expression class in the CK+ dataset

Table 5 shows only two images that were misclassified by one of the EfficientNet-B0 models and reliability of the model in making predictions built with the transfer learning approach proposed by the researchers. The first image belongs to the label of the type of expression, but the model predicts as an expression of disgust. In this case, the model of difficulty in discriminating images of sadness is almost like the expression of disgust specific to the facial expressions of the person presented in this Table 5, but visually the expressions of sadness and disgust have almost similarities such as the eyebrows wrinkle down or sometimes upwards and the lips tend to shrink, the teeth are closed, and the skin of the face looks wrinkled. The second image belongs to the label of the type of surprise expression, but the predictive model is included in the expression of sadness. Why this can happen is because specifically for a person's face in this second picture, when surprised the model predicts looking sad because his eyebrows seem to express sadness, but visually the expression of surprise and sadness has a slight difference that lies only in the state of the eyebrows tending upwards to widen and the skin wrinkles upwards.

Table 5. Misclassified images from the CK+ datasets

Misclassified Image: True Class → Predicted Class
Samples From CK+	biao_5a.png	biao_5b.png

In the analysis of Table 5, it can be concluded that the state of expression of each person tends to be different and is influenced by certain skin color, age level, and ethnicity. In other typical, when two people have the same expressions, it is not necessarily that the two people have expressions that look the same visually since everyone has a unique character in each of their expressions, although most of each expression can be classified. In certain cases, for example, visually expressing fear but the model predicts the person is being surprised, and the human being also has difficulty in predicting the expressions of everyone. 

4.5 Results comparison with existing method

Using the CK+ and JAFFE datasets, this section compares the proposed FER method's performance to that of existing leading expression recognition methods. Table 6 also includes the separation of training and test data, as well as the specific properties of the individual methods, for a better understanding of the accuracy of test set recognition. The analysis incorporates both traditional and deep learning-based methods. Most existing techniques make use of the JAFFE dataset, which has only 213 samples. The CK+ dataset, on the other hand, is significantly larger, with 1308 images containing frontal views. It should be noted that the front view image is easier to classify than the front view and profile image. In the existing studies, various strategies for separating training and test samples were used, as shown in Table 6. Furthermore, the comparison table's presentation of the significance of each method with the technique (used in feature selection and classification) is very useful for understanding transfer learning technique proficiency.

Table 6. Comparisonxofxthexaccuracy of proposed method with existing work on CK+ and JAFFE datasets