A Comparative Approach for Facial Expression Recognition in Higher Education Using Hybrid-Deep Learning from Students' Facial Images

Deep learning-based facial expression recognition has potential applications in social media, marketing, public safety, and human-computer interaction. It is thus regarded as an advanced technique worthy of further development. The aim of this study is to develop techniques that enable facial recognition to be automated for eight mood classes: attention, anger, disgust, fear, happiness, neutrality, sadness, and surprise expressions to monitor students’ expressions during online lectures. Most recent research in facial expression recognition has focused on seven mood classes: anger, disgust, fear, happiness, neutrality, sadness, and surprise expressions [1, 2]. Since student engagement is very important in education, "attention" has been added to emotional states in the study. A summary of research related to these techniques is given below.

Hussein et al. [1] relied on geometric aspects of facial expressions with a modest level of complexity in their study. They employed three face image databases: VDMFP, BINED, and MMI. These databases hold images of various facial expressions, captured under various imaging situations, which were first rotated around a central point, then analyzed with deep learning to extract features. Their method achieved 94% for accuracy. Sajjad et al. [2] proposed a method which can provide information on potential suspicious activity using facial expression recognition. They used Raspberry Pi to carry out this purpose, where the video is captured by the Raspberry Pi camera, and faces are detected using the Viola-Jones algorithm. The Gabor filter is also used for the pre-processing in face region and the Oriented Fast and Rotated Brief (ORB) algorithm is used for the feature extraction. SVM is used as a classifier, where SVM classified the facial expressions (anger, disgust, fear, happiness, neutrality, sad, and surprise). By the classification of the facial expressions, the emotion behind the scene was predicted. Based on these predictions, if hostility is detected steps can be taken to avert negative outcomes. They used public databases (CK+, JAFFE, MMI) for testing and training stages and their model achieved 92% for accuracy.

Leo et al. [3] suggested a system that uses a robot to calculate the robot's interaction with children with autism spectrum disorders (ASD). Researchers provide the robot with the facial expression for its imitation, then the robot asks the child to imitate it to assess his or her emotional imitation abilities and quantify the time-delay between the robot's request and the child's response. Faces are cropped and registered, and the HOG descriptor is used to create a data vector that is fed into SVM, resulting in an estimate of the observed facial expression. The researchers achieved 94% for the F1-Score. Syazana-Itqan et al. [4] developed a face recognition system with a MATLAB-based Convolution Neural Network (CNN) and a Graphical User Interface (GUI) as the user interface. To minimize neural network training time, the suggested CNN can accept new data by training the last two layers out of four layers. They used images from the AT&T database and the JAFFE database, with 40 subjects from AT&T and 10 from JAFFE database and achieved 100% for accuracy.

Mayya et al. [5] developed an approach based on a Deep Convolutional Neural Network (DCNN) to automatically distinguish facial expressions through identification of an individual’s facial expression using a single image. Using a JAFFE Dataset for a general-purpose graphic processing unit (GPGPU) significantly reduced the feature extraction time. Images from CK+ Dataset and JAFFE Dataset (Japanese Female Facial Expression) were used for several facial expressions such as anger, disgust, fear, happiness, neutral, sadness, and surprise. This method achieved 96% for F1-score. DNNs based on CNNs were employed by Fathallah et al. [6] for Automated Facial Expression Recognition. To improve results, they use the Visual Geometry Group model (VGG) to fine-tune their architecture. They test their design with a number of large general databases (CK+, MUG, and RAFD) in order to assess it. Their findings demonstrate that the CNN method is relatively effective in image expression identification on various public databases, indicating that facial expression analysis has improved. Their model achieved 93% for accuracy. Sang et al. [7] proposed DCNNs that can effectively interpret semantic information accessible in faces automatically without hand-designing feature descriptors, based on current advances in deep learning. They used the Kaggle dataset’s competitor, the FERC-2013 dataset, which contains 35,887 grayscale images with a resolution of 48x48 pixels. These images have been separated into three categories by Kaggle: 28,709 training images, 3589 public test images, and 3589 private test images. Sang et al.’s method achieved 73% for accuracy.

Tarnowski et al. [8] used the KDEF dataset for emotion recognition using facial expression, focusing on seven emotional states (neutral, joy, sadness, surprise, anger, fear, disgust). The features are classified by using K-NN and MLP neural network classifiers and their model achieved 73% for accuracy. Qayyum et al. use JAFFE and CK+ dataset and MS-Kinect dataset in their study. They use Stationary Wavelet Transform (SWT) to extract features for facial expression recognition. In order for the stationary wavelet to transform properly, a combination of horizontal and vertical sub-bands are used, where these sub-bands contain information on muscle movement for the majority of the facial expressions. Their method achieved 94% for accuracy [9].

Lopes et al. [10] propose a simple approach to facial expression recognition with little data. In the method that they proposed, a combination of a CNN and specific image pre-processing steps (rotation correction, cropping, down-sampling, and intensity normalization) is used. The pre-processing steps are conducted using OpenCV and CNN library (CAFÉ). In addition, they used the public databases (CK+, JAFFE, and BU-3DFE) for training and testing stages. Their model achieved 95.75% for accuracy.

Surace et al. [11] suggest a method for group emotion recognition in the wild modelled on a combination of deep CNNs and Bayesian classifiers. They used the TensorFlow library to train CNNs and implemented the proposed approach in Python. Furthermore, OpenCV was used to perform pre-processing operations on their dataset images, where they used the Emotion Recognition in the Wild 2017 (EmotiW17) as a database. Their method achieved 67.75% for accuracy.

Gupta et al. [12] relied on the size of the face and eyes and on deep learning for facial detection. They used OpenCV for the identification of the face, as OpenCV is considered a classifier that helps in image processing. In the later stages of the processing operations, a deep belief network was used. All these operations used the Python program and their model achieved 97% for accuracy. Arya and Agrawal [13] introduced a face recognition approach based on Deep Learning and Linear Discriminant Analysis (LDA), in which typical image recognition system architectures include three primary stages. In the first stage, referred to as pre-processing, the facial images are partitioned into various face components. Prior to the LDA algorithm being applied, the feature extraction process is conducted. In the final step, the neural network is proposed for training on the extracted face classes and features and the model generated is used for face recognition. Li and Lam [14] developed a technique using a Deep Neural Network (DNN) to recognize facial expressions. The Gabor filter is used to extract facial expressions, and then kernel PCA is applied, after which the data extracted from the previous stage is processed into the deep neural network. Their model achieved 91% for accuracy.

Carcagnì et al. [15] used CK+ dataset, for facial expression recognition. One method is proposed, based on histograms of oriented gradients (HOG), where HOG is a descriptor. After obtaining the features of HOG, these are provided to inputs of a group of Support Vector Machines (SVMs), where SVM is a discriminative classifier. They achieved 92.9% for F1-Score.

Chen et al. [16] have proposed a softmax regression autoencoder network (SRDSAN) for facial expression recognition to address the dual challenges of learning efficiency and computational complexity, where a Deep Sparse Autoencoder Network (DSAN) is used to extract high-level features and learn facial emotion features and the facial emotions are classified by using softmax regression (SR). The researchers used (CK+, JAFFE) as databases and the method proposed is to combine with robots in order to analyze and understand human emotions. Their method achieved 89% for accuracy. Rao et al. [17] have proposed a method to combine the features of facial expressions and speech on the Indian Face Database and Berlin Speech Database. They used MFCC (Mel Frequency Cepstral Coefficients) for recognizing speech, and MSER (Maximally Stable Extremal Regions) for facial emotion recognition. Also, SVM classifier was as used for emotion classification and AdaBoost for gender classification. Their model achieved 92% for accuracy. Tonguç and Ozaydın Ozkara [18] analyzed pictures of facial expressions of 67 students during practical and theoretical lectures using software developed through Microsoft Emotion Recognition API and C# programming language. After analysis, they found that the feelings of disgust, sadness, happiness, fear, contempt, anger, and surprise changed during the lecture depending on the way the lecturer presented the lecture. Xu et al. [19] proposed a method for estimating the positive or negative state of focus based on the head position. They used CNN networks with two public databases CK+ & BU-4DFE and their method achieved 91.5% for accuracy. Wu et al. [20] developed a Weight Adapted Convolution Neural Network (WACNN)-based technique for facial expression recognition, where the pre-processing for the facial expression images is conducted via geometry normalization and gray normalizing. Following this, PCA is used to extract the low-level expression feature, then WACNN is used to learn representative features and recognize them, resulting in facial expression information. As databases, they used (CK+, JAFFE, and SFEW2.0) and their method achieved 94% for accuracy. Wang et al. [21] developed a system to recognize speech emotion and facial emotions, they used the CNN and RNN to distinguish the facial expressions, and they used LSTM and CNN to recognize speech emotion, they employed a variety of datasets such as RML, AFEW6.0, and eNTERFACE’05 for achieved those purposes. Li et al. [22] proposed a method for recognizing facial expressions using a convolution neural network (CNN) to accurately separate the salient regions from a face image. Following that, the Gaussian Markov random field (GMRF) model was improved to improve texture features' ability to represent image information, and a novel feature extraction algorithm called specific angle abundance entropy (SAAE) was designed to improve shape features' ability to represent image information. The texture and shape features were then combined and trained and classified by the support vector machine (SVM) classifier.

Examining the past research that has been summarized, all the research is very recent and valuable, and addresses multiple goals. However, few of these previous studies are focused on enhancing the education in universities. Furthermore, in terms of the applicability of existing research to online education contexts, limitations can be identified. Firstly, most of the previous research focused on recognizing the facial expressions of seven emotions (anger, disgust, fear, happiness, neutrality, sadness, and surprise) without the inclusion of an expression to measure engagement. In addition, most of the research to date has used public data sets from the internet such as CK + and others. This led us to aspire to access new techniques that will enable facial expression recognition to be automated for eight mood classes, rather than seven: anger, disgust, fear, happiness, neutrality, attention, sadness, and surprise expressions. “Attention” was added to the facial expressions because this is important in an educational context as a measure of engagement. A Deep learning-based CNN and image processing techniques were employed.

The database was trained on the pre-trained networks such as Alexnet, GoogleNet, ResNet18, ResNet50, MoblieNetV2, and VGG16. A database for the current research was created from the images of the faces students at one university. These images express the facial expressions of the eight expressions, which are mentioned above.

In the current study, image processing, deep learning toolbox, and transfer learning algorithm are used in Matlab 2021A where modifications are made to the last layers in the Convolutional Neural Networks, and then these networks are trained on the new data. If the classification accuracy is sufficient, then the desired accuracy is obtained. However, if the classification accuracy is insufficient, modifications will be made to the Convolutional Neural Networks, and retraining on that data will be conducted until the desired sufficient accuracy is obtained (see Figure 2).

2b.png

Figure 2. Workflow of transfer learning

A computer with an i7/7th CPU, 16GB RAM, and a 4GB GTX1050 was employed in this study. GPU/CUDA can be used for processing acceleration throughout the training and testing phases thanks to these features. While the NVIDIA CUDA® Deep Neural Network (cuDNN) library is a GPU-accelerated library of deep neural network primitives, it also includes highly fine-tuned implementations of standard processes including foreground, background, convolution, pooling, normalization, and activation layers (see Figure 3).

Our dataset has been applied to six pre-trained CNN networks and transfer learning and K-cross validation. Those networks are AlexNet + TL + KFCV, GoogleNet + TL + KFCV, ResNet 18 + TL + KFCV, ResNet50+TL + KFCV, MobileNetV2+ TL + KFCV, and VGG16+TL + KFCV. The transfer learning (TL) + 5-fold validation process was applied to the previous six trained deep learning architectures (AlexNet, GoogleNet, ResNet 18, ResNet 50, MobileNetV2, and VGG16). The features extracted from the training images were used to train the models to obtain the better parameterization of these architectures. By using the transfer learning algorithm, the six networks are trained on our data and thus have new parameters, and the old parameters are removed. Our dataset consists of images of the faces of university students and the number of images is 6720 pictures distributed over eight categories of facial expressions (anger, disgust, fear, happiness, neutrality, sadness, attention, and surprise) for each category of 840 images. The practical experiments were performed separately on the six networks. Table 1 shows the training and testing time for each network. It can be noted that the time of the training and testing is related to the maximum number of parameters that are distinguished by each network from the other (Table 2).

Table 1. Time of the stages of the training and testing for various CNN Networks using GPU

Table 2. The maximum number of parameters for various CNN networks

*On a path from the input layer to the output layer, the network depth is defined as the greatest number of sequential convolutional or fully connected layers.

5.1 Performance metrics

It can be inferred that many values help in evaluating the performance of the model since these values are inferred from the confusion matrix which is the cornerstone of evaluating a classification model (that is, classifier). This square matrix displays the performance of a learning algorithm by reporting the counts of the True positive (TP), True negative (TN), False positive (FP), and False negative (FN) predictions of a classifier. In the current study, confusion matrices for the models are shown (AlexNet + TL + KFCV, GoogleNet + TL + KFCV, ResNet 18 + TL + KFCV, ResNet50+TL + KFCV, MobileNetV2+ TL + KFCV,and VGG16+TL + KFCV) on which our datasets were trained (see Figures from 13 to 18).

13.png

Figure 13. Confusion Matrix for AlexNet+TL+KFCV

14.png

Figure 14. Confusion Matrix for GoogleNet+TL+KFCV

15.png

Figure 15. Confusion Matrix for ResNet18+TL+KFCV

16.png

Figure 16. Confusion Matrix for ResNet50+TL+KFCV

17.png

Figure 17. Confusion Matrix for MobileNetV2+TL+KFCV

18.png

Figure 18. Confusion Matrix for VGG16+TL+KFCV

From the confusion matrix, several parameters of performance metrics can be deduced which are given as follows:

1. Accuracy (ACC) measures how many times a classifier makes an accurate prediction, and is calculated using the following equation:

ACC= (TP+TN)/(TP+TN+FP+FN)

2. Recall (REC) is also called sensitivity (SNS) or true positive rate (TPR). It estimates the proportions of true positives from all the positive values observed for a target:

SNS=TP/(TP+FN)

3. Precision (PRC) estimates the proportions of true positives that were precisely defined, and is calculated as following:

PRC=TP/(TP+FP)

4. Specificity (SPC) estimates the proportions of true negatives from all negative values observed for a target, also known as true negative rate (TNR).

SPC=TN/(TN+FP)

5. F1 Score shows the harmonic mean of Precision and Recall values. F1 score is also a sound measure to estimate an imbalanced classifier.

F1-Score=2xPRCxSNS/(PRC+SNS)

Now, based on the confusion matrices, it is possible to deduce performance metrics (ACC, SNS, PRC, SPC, F1-Score) for the six networks that are used in this research (AlexNet + TL + KFCV, GoogleNet + TL + KFCV, ResNet 18 + TL + KFCV, ResNet50+TL + KFCV, MobileNetV2+ TL + KFCV, and VGG16+TL + KFCV). Tables have been organized showing the values of performance metrics Tables 3-8).

In order to compare the results between the six studied pre-trained networks, the average values of the evaluation parameters for each network were separated. These values can be represented in a chart (Figure 19) in order to facilitate the comparison of the results.

Table 3. The values of the performance metrics for AlexNet

Table 4. The values of the performance metrics for GoogleNet

Table 5. The values of the performance metrics for ResNet18

Table 6. The values of the performance metrics for ResNet50

Table 7. The values of the performance metrics for MobileNetV2

Table 8. The values of the performance metrics for VGG16

Because of their importance, AUC-ROC curves are included in this research (Figure 20). The AUC-ROC curve helps visualization of the performance of the CNN networks classifiers used in this research, and the ROC curve summarizes the performance of the classifier across all possible thresholds. The graph of the ROC curve is plotted with the true positive rate (TPR) in the y-axis and the false positive rate (FPR) in the x-axis for all possible thresholds.

AUC: AUC is the area under the ROC curve. If the classifier is great, the true positive rate will increase, and the area under the curve will be close to 1.

By examining the confusion matrices (Figure 13 to 18, Tables 3 to 8, and Figures 19 and 20), it is clear that the best results were recorded by ResNet18 closely followed by ResNet50. The average values of performance metrics (ACC, SNS, PRC, SPC,F1-Score) for ResNet18+TL+KFCV are 0.997, 0.989, 0.989, 0.998, 0.989 respectively while for ResNet50+TL+KFCV are 0.996,0.986,0.986,0.998,0.986 respectively. Next came the results for MobileNetV2 + TL + KFCV, AlexNet + TL + KFCV, and VGG16 + TL + KFCV, respectively, while the lowest results were recorded by GoogleNet+TL+KFCV.

19.png

Figure 19. The comparison of averages the performance metrics for six deep learning methods that were used in this study

In addition, by examining ROC curves and AUC values (Figure 20) the best results were recorded when the pre-trained network of ResNet18+TL+KFCV was used, and the same result was obtained for ResNet50+TL+KFCV. This means that they scored the best results for automatic recognition, of the facial expressions of anger, disgust, fear, happiness, neutrality, sadness, attention, and surprise.

From the tables AlexNet, MobileNetV2, and VGG16 (Tables 3, 7 and 8) it can be noted that values of F1-Score for facial expressions of anger, disgust, fear, happiness, and surprise were high. However, for the facial expressions of neutrality, attention, and sadness, they decreased by approximately 5% to 7% from the highest value recorded.

As for GoogleNet (Table 4), it can be noted that values of F1-Score for facial expressions of anger, disgust, fear, happiness, and surprise were approximately 85%, but for facial expressions of neutrality, attention, and sadness, they decreased by approximately 20% from the highest value recorded. If we look at the values of F1-Score (Tables 5 and 6) for ResNet18 and ResNet50, the scores for facial expressions of anger, disgust, fear, happiness, and surprise were very high, but they decreased by approximately 2% for facial expressions of neutrality, attention, and sadness.

It is possible that the facial expressions of neutrality, attention, and sadness may differ from one person to another, and it may be difficult to recognize the state of attention and the neutrality state of a person, so the low F1- Score values for these facial expressions can be noted.

From all the above, it is evident that the best results were recorded by ResNet18 and ResNet50. By comparing ResNet18 and ResNet50 for the time of the stages of the training and testing examination It can be seen that less time was recorded by ResNet18 (see Table 1). It may be the recommended approach that automatically tracking the students' emotional states throughout the lecture, gives the lecturer immediate feedback and helps to improve the educational experience.

20.png

Figure 20. ROC-AUC Curves for classification in the six CNN models

5.2 Comparison of the current study with other studies

In this study, the database of our own was created from the images of the faces of 70 students from the faculty of Engineering expressing eight facial expressions (anger, disgust, fear, happiness, neutrality, sadness, attention, and surprise). The number of images of the students’ faces was 6720, distributed equally over the eight facial expressions mentioned above, that is, for each category 840 images. Most of the existing studies used public databases and did not have private data such as that used in our research. Most of the existing studies focused on facial expressions in six or seven categories, while our study focused on facial expressions in eight categories. In addition, deep learning in our research was used with transfer learning with k-fold cross-validation. The results obtained were compared with similar studies on facial expression recognition and a summary of this comparison is shown in Table 9. From the table, it is clear that the results of the current study give high accuracy and AUC values. Furthermore, the obtained results in the current study for the performance metrics (Sensitivity, Specificity, Precision, and F1-Score value) are high. From this comparison, it can be seen that this study can contribute to the field of recognition of facial expressions for students for educational purposes.

Table 9. Comparison of the current study with other studies which used various deep learning methods for facial expression recognition