Analysis of Face Emotion Identification and Recognition Using State-of-the-Art Deep Learning Models

An important part of interpersonal communication, emotions are basic components of human nature.  Humans use body language, facial gestures, and voice as nonverbal ways to convey their emotions. Among the many components of Emotion Detection, facial expression analysis has received the greatest amount of attention and research. Recognising emotions from facial expressions has recently attracted the attention of the fields of psychology, psychiatry, and mental health. The ability to automatically identify emotions based on facial expressions is crucial for many applications, including smart homes, healthcare systems, the diagnosis of emotional disorders in individuals with autism spectrum disorders and schizophrenia, human-computer interaction (HCI), human-robot interaction (HRI), and social welfare programs that leverage HRI. Researchers have taken an interest in FER due to its potential usefulness in a variety of contexts [1].

The fundamental objective of FER is to assign emotional states to different facial expressions. There are two primary processes in the traditional FER: emotion recognition and feature extraction. The images need to be further processed by resizing, cropping, and generally improving their quality. Face detection involves cropping the face area after removing the background and non-facial regions. The primary function of a conventional FER system is to feature extraction by a processed picture; the techniques now in use include DWT, LDA, etc. [2-4]. Ultimately, by categorizing the retrieved features, typically employing an NN and other ML approaches, emotions are understood [5, 6]. After that, the collected features are sent to ML classifiers, like KNN [7], HMM [8], SVM [9], DT, and NB [10]. The key obstacles to enhancing the system performance of traditional vision-based FER include labor-intensiveness, error-proneness, dependence on domain experts for critical feature selection and classification, and independent feature extraction and classification [11]. In order to provide somewhat greater accuracy, the researcher looked into DL-based FER techniques. Several researchers used CNN-based techniques after being impressed by the recent success of DL approaches [12-15].

Recently, DNNs, especially CNNs, have been more interested in FER because of their integrated feature extraction approach from images. Several efforts to address FER issues have been reported by CNN [16-20]. Nevertheless, the CNN with only a few layers was taken into account by the extant FER methods, even though its deeper model is significantly more effective at other image-processing tasks [21]. The obstacles associated with FER may be the cause of this. A picture with a decent amount of resolution is first required for emotion identification to examine data with a lot of dimensions. Secondly, the classification task is ultimately complicated by the extremely low difference in features that results from varying emotional states. Conversely, an exceedingly deep CNN is composed of an immense quantity of concealed convolutional layers. In the CNN, the process of training an enormous quantity of concealed layers becomes tedious and fails to generalize effectively. Moreover, accuracy does not increase beyond a certain threshold as a result of the vanishing gradient problem, which is exacerbated by the mere addition of layers [22, 23]. In order to improve accuracy, many changes and methods are implemented to a DCNN architecture and training method [24-26]. A widely employed pre-trained DCNN models are VGG-16 [27], Resnet-50 [28], Resnet-152 [29], Inception-v3 [30], and DenseNet-161 [31]. Although deep models may be trained, it requires a substantial amount of data and computational capacity. This project aims to examine FER models that use DL, CNN, and TL [32], and secondly, to develop a superior model. After improving the overall performance of a DL model, this work builds a model for FER using CNNs that have been tuned for hyperparameters. The dataset FER2013, sourced by Kaggle, has around 32,298 grayscale pictures depicting diverse facial expressions on humans. Anger, fear, disgust, sadness, happiness, surprise, and neutral are seven different categories of emotions into which labelled data is classified using the Multiclass classifier. The CNN algorithm will be used to conduct emotion identification on each of the two sets, the training and testing sets of a classified dataset. The veracity of a model with respect to the FER2013 dataset will be assessed based on the results obtained. The term "training set" is used to describe the dataset or collection of samples utilized to train a model. The test set, sometimes called a subset or collection of samples, is employed to evaluate how well the trained model performed.

1.1 Research motivations and contribution

The increasing need for efficient and successful FER systems, which have a wide range of uses in fields including marketing, healthcare, security, and HCI, is what spurred this study. Despite the advancements in deep learning, there remain significant challenges in achieving high accuracy and robustness in FER models across varied environments and datasets. While CNNs have demonstrated promising results, the potential of Transfer Learning (TL) models—especially pre-trained models—has not been fully explored for FER tasks. This study is motivated by the need to explore and compare these two approaches to identify the most suitable method for real-world applications, offering insights that could lead to more accurate, scalable, and adaptable FER systems. The following research contribution of this work as:

• This work aims to create a DL model capable of reliably identifying a human face's mood using TensorFlow analysis of facial aspects through experimental research.

• Creating a CNN model-based FER technique that effectively handles problems using transfer learning.

• Create a deep learning model utilizing the FER2013 dataset, hyper-parameters, and CNN.

• Data balance was performed on the pre-processed data. Following validation on the test set, the DL model was trained using the train set.

• I Improving the performance of matching facial expression classification and identification is achieved by continuous adversarial training of CNN, which allows for deep extraction of features from the processed expression dataset.

• A proper training size dataset and hyper-parameter adjustment using CNN are also part of the experiment to enhance performance.

• An evaluation of the suggested method's efficacy in relation to existing emotion recognition techniques.

1.2 Novelty of this work

The novelty of the research lies in its comprehensive comparison of CNN and TL models for Facial Emotion Recognition (FER). While both approaches are well-explored in the domain, the paper differentiates itself by evaluating the performance of these models on multiple datasets, thus providing deeper insights into their generalization ability across different FER tasks. Furthermore, the study innovates by incorporating various pre-trained models in transfer learning, contrasting them against traditional CNN architectures, and analyzing the trade-offs in accuracy, computational efficiency, and real-world applicability. This comparative evaluation contributes to optimizing FER systems by offering a detailed understanding of how these methods can be leveraged for better performance in diverse environments.

FER, a subfield of computer vision, involves the complex task of identifying emotions from facial images. The suggested technique uses the FER2013 dataset as its main source of data collection in an effort to improve FER's accuracy. Following this, data preprocessing techniques like grayscale conversion, normalization, and resizing are applied to enhance image quality and consistency. The processed data is then split into training, validation, and testing sets to facilitate robust model training and evaluation. In the classification stage, a transfer learning (TL) model is utilized, enabling the reuse of pre-trained deep learning architectures to effectively perform FER. The final step is to evaluate the model by measuring its performance using metrics such as the F1 score, recall, accuracy, and precision. The entire methodology is structured into primary stages, as illustrated in Figure 1.

1.png

Figure 1. Flowchart for identification of facial emotions

3.1 Data collection

An initial stage entails the acquisition of input images by a FER 2013 dataset, a dataset that was expressly developed for FER in 2013 [30]. The reason this dataset was chosen is that deep networks need a large database to be trained effectively. When the input picture changes lighting, emotion, or attitude, the Gabor filter is used to fix the local distortions.

There are around 32,298 images in this collection, and each one represents one of seven different emotions: joy, wrath, surprise, fear, disgust, sorrow, or neutrality. There are headshots in both staged and candid poses. Figure 2 displays a selection of photos taken from the FER2013 database.

2.png

Figure 2. Sample images from FER2013 [30]

3.2 Data pre-processing

A dataset called FER2013 was used for the thesis. Nearly 32,298 grayscale pictures of human face expressions are part of the dataset, which is derived from Kaggle. These images are categorized into 7 categories: anger, revulsion, happiness, fear, sorrow, astonishment, and neutral. Each image in a database has been size-normalized to 48×48 pixels, and it is exclusively grayscale. Instances are viewed using the head method, and the Pandas library is employed to import the dataset.

In data preprocessing, abnormalities such as missing or incorrect numbers, illogical data structures, or data types are removed from the dataset. This generated high-quality data of high quality and facilitated the execution of an efficient data analysis. Before normalization, Figure 3 displays a bar chart including all 7 emotion classes found in a FER2013 dataset. A dataset's x-axis shows the total number of classes, while a y-axis displays the sample size. Therefore, we have resampled the whole dataset and fairly distributed 8,500 image examples across all of the classes. Following data normalization, Figure 4 shows a bar chart depicting the 7 emotion classes contained in the FER2013 dataset.

3.png

Figure 3. Bar graph of the FER2013 dataset’s seven emotion classes

4.png

Figure 4. Bar graph of the FER2013 dataset’s seven emotion classes after data balancing with 8500 samples in each emotion class

3.3 Dataset train-test split

The scikit-plot package and the train set divide function were employed to divide an entire dataset into 7 train sets and seven test sets of varying sizes. We considered training set sizes between twenty percent and eighty percent. Training the DL model using varying sized training sets, beginning at (20, 30, 40, 50, 60, 70, 80) percent of the complete dataset, was the next stage. A training set size that produced the best accuracy for the DL model was identified by analysing the many results that were obtained for every training set size.

3.4 Classification with CNN

One of the most well-known types of DNNs is the CNN. Convolution derives its name from the mathematical operation of linearly connecting matrices. The assessment of visual pictures makes extensive use of this kind of ANN. In NLP, recommendation systems, classification, segmentation, medical image analysis, and video and image identification, they play an important role in brain-computer interfaces. Impressive achievements were achieved by picture data applications, including computer vision, NLP, and the biggest image classification dataset (Image Net) [12].

What a magnificent design CNN has. An input layer, two hidden layers (the convolutional and pooling layers, and an output layer, respectively), and a CNN are displayed in Figure 5 [13]. After training, the model receives input from the user, and the CNN processes it to provide assessed outputs, as shown in the figure.

5.png

Figure 5. Schematic diagram illustrating the architecture of basic CNN [14]

3.4.1 Layers of CNN

Following is a list of the three primary CNN layers (concerning Figure 5 up top):

(1) Convolutional Layer

The input pictures' various attributes are first extracted by the convolutional layer. The mathematical procedure of convolution with an input image is performed by a filter of size M×M (where M denotes the height and width of the square filter, such as 3×3 or 5×5) in this layer. As it moves across the input picture, the filter calculates a dot product with the relevant parts of the image.  This process culminates in the creation of a feature map that draws attention to critical picture elements like edges and corners. Subsequent layers use this feature map as a foundation for learning a wide range of attributes from the original image.

(2) Pooling Layer

After a convolutional layer, there comes the pooling layer. Reducing the computational expenditure by shrinking the convolved feature map is the main goal of this layer. To do this, we work on each feature map separately and reduce the connections between layers. There are several kinds of pooling operations that are defined by the methods utilised.

Max Pooling states that the feature map produces the largest element. By utilising average pooling, one may ascertain the average of every element inside a pre-sized picture segment. Min Pooling takes the feature map and pulls out the smallest part. By combining the elements in the specified area, Sum Pooling determines their total value. Combining the Convolutional and Fully Connected (FC) Layers is a typical use case for the Pooling Layer.

(3) Fully Connected Layer

To connect neurons in other layers, the FC layer uses weights and biases in addition to neurons. The output layer often follows the final few layers in a CNN. The data becomes progressively more complicated beyond this point because of the large number of hidden layers with different weights for every neuron's output. This is where all the data processing and reasoning happens.

Lastly, input pictures are transferred to the FC layer after being flattened from the layers above. After that, the mathematical functional operations are often carried out by passing the flattened vector through a few additional FC levels. The classifying procedure starts at this point.

3.4.2 Hyper-parameter tuning of CNN model with fine-tune

One of ML's hyperparameters that may influence how the model learns. Various additional parameters, including node weights, are set throughout the training process. For various datasets and models, a model's efficacy can be enhanced by utilizing a variety of hyperparameters. This set of hyperparameters is for optimizing a model, as the name suggests [15].

(1) Learning Rate (0.001): The value of this hyperparameter dictates the weight that newly acquired data will carry over older data. A failed optimisation might occur if this hyperparameter's value is significant, as the model would potentially pass over the minimum. Conversely, a slow learning rate will cause convergence to take longer than expected.

(2) Batch Size (32): An instruction set is broken down into numerous sections to make learning it faster. There will be an increase in the learning time and the need for additional memory to perform matrix multiplication if the group size is greater than. The error computation will contain a greater amount of noise if the group size is reduced.

(3) Number of Epochs (100): The epochs in DL stand for whole data cycles that can be learnt. Epochs are particularly important throughout an iterative learning process. Assuming the validation error drops, increasing the number of epochs is acceptable. Reducing the number of epochs is recommended if a validation error does not change after a certain number of epochs. It's frequently called "early stopping."

(4) Number of Layers: The performance of CNN models is enhanced by adding more layers.

To facilitate generalization, we employed dropouts at regular intervals, as this is a CNN model. The ELU was selected as an activation function in a Batch Normalization procedure. This activation function is based on ReLU and is responsible for determining the smoothness of the function when the inputs are negative. Its superior performance in this case and its ability to fix the ReLU expiration problem make it the preferred choice over LeakyReLU. Also chosen for use is the 'he_normal' kernel initializer, which is more suited for ELU. We ensured that we utilised a reduced amount of dropout, as dropout typically contributes to cacophony. The padding value was assigned to 'SAME' during the model training process, as the input and output sizes are identical.

3.5 Justification

The choice of methods and parameters in the development of the CNN model appears to be based on established practices in deep learning. For instance, the use of the FER2013 dataset is justified by its size, diversity, and suitability for training deep networks, given their reliance on large datasets to learn effectively. The decision to employ data preprocessing, such as normalization and resampling to balance class distributions, ensures uniform representation of all emotion classes, which enhances the model's generalization ability. Furthermore, the selection of hyperparameters, including a learning rate of 0.001, batch size of 32, and 100epochs, reflects a standard approach aimed at optimizing convergence speed and minimizing overfitting. The use of ELU as an activation function, combined with the "he_normal" kernel initializer, addresses challenges like the vanishing gradient problem and ensures better initialization for deeper networks. Similarly, incorporating dropouts and batch normalization fosters model regularization and training stability.

The selection of CNN over advanced models like EfficientNet or Vision Transformers (ViTs) is justified by its simplicity, computational efficiency, and suitability for the FER2013 dataset. Using optimised hyperparameters such as a learning rate of 0.001, batch size of 32, ELU activation, and the 'he_normal' initialiser, CNNs succeeded in achieving greater performance when it came to extracting localised features, which are vital for emotion identification. Techniques like dropout regularization and batch normalization further improved generalization and stability. These factors highlight CNN’s effectiveness for this task, while advanced architectures could be explored for potential future improvements.