Micro-Expression Recognition by Using CNN Features with PSO Algorithm and SVM Methods

The models in the ME studies consist of three parts: preprocessing, feature extraction, and classification. The preprocessing phase consists of separating the non-face area and detecting, aligning, and cropping the face region to prevent the effect of head movement [9]. In studies on face detect, DRMF method [10, 11] Active Shape Model (ASM) 68-point method [4, 8, 12] were used. Dlib machine learning toolkit of the OpenCV library used for both detection and alignment are used [9, 13, 14].

Methods used as Feature Extraction are divided into traditional methods and data-oriented methods; these conventional models are Optical Flow, LBP, and its derivatives. Optical flow is a technique that detects motion variation between two video frames [9, 15-17]. Local binary patterns (LBP) are used to extract texture features from gray images [4]. The derivative of LBP, which encodes both spatial and temporal information, is used LBP-TOP [5, 8, 18, 19].

Deep learning methods are another method used in ME called data-oriented methods. In addition, using deep learning architecture from raw images, both feature extraction and classification can be done [20, 21]. Transfer learning methods have been used to overcome the problem when data is scarce [10, 20, 22]. Another method used in classification is SVM and its derivatives which are of great importance for machine learning [2, 3, 5, 8].

Based on the scarcity of datasets in ME studies and the idea that CNN-based studies in this area are limited [23], a hybrid model with traditional and data-oriented is proposed as our motivation in the study. In addition, CASME-II, SAMM, SMIC datasets, which are the most widely used and publicly available spontaneous micro-expressions, were used in our study.

1.png

Figure 1. ME classification framework architecture

4.1 Preprocessing

In the proposed study, Apex single frame images were taken from ME dataset frames. The Apex frame was chosen because it has the highest ME density in a video frame sequence [30]. While the index of this frame is provided in Casme-II and SAMM datasets, it is not provided in the SMIC dataset, so the D&C-RoIs (Divide and Conquer-Region of Interest) [31] technique, which automatically finds its approximation position, has been developed. Because the D&C-RoIs technique provides successful performance in apex detection, it has been used in ME studies [6, 25, 32]. The D&C-RoI method includes the feature descriptor LBP and divide&conquer techniques. LBP calculates the features of the face regions of each frame in a video sequence. Then, the feature difference of each frame is calculated by taking the onset frame as a reference. The divide & conquer algorithm determines the position (index) of the apex frame which has the maximum difference.

The Apex frame was obtained by calculating the RoI regions in each frame and calculating the correlation between the first frame and other frames with the LBP descriptor [31]. The calculation of the correlations of the first frame and the remaining frames using LBP is given in Eq. (1) [25].

$d=\frac{\sum_{i=1}^B h_{1 i} x h_{2 i}}{\sqrt{\sum_{i=1}^B h_{1 i}^2 x \sum_{i=1}^B h_{2 i}^2}}$            (1)

here, h₁ is the first frame, h₂ is the other frames, and B is the number of boxes in the h₁ and h₂ histograms. The difference ratio of LBP features (1-d) between the eyebrow, eye, and mouth, the three most effective ROIs in ME recognition, is compared. The ROI with the highest difference ratio is selected. Finally, a divide-and-conquer strategy is applied [25, 31] to seek frames that have maximum facial muscle changes. According to the divide and conquer strategy, a video clip frame sequence is divided into subsequences. The correlation coefficients of the frames are summed in each sub-array. The index with the largest sum is kept, and the rest is discarded. It continues until the frame with the maximum value is found [2].

In Figure 2, the positions of the frames in a video sequence in the micro-expression and their optical flow data of these positions are seen. This example is the happiness sample found in the Casme-II dataset.

2.png

Figure 2. Onset, apex and offset in ME video sequence

Since CASME-II and SMIC samples have preprocessed formats, they were not required preprocessing. However, since SAMM samples did not go through any preprocessing step, preprocessing was performed on the SAMM dataset before extracting the optical flow properties of the images. Face detection, alignment, cropping, and recording processes were applied to the samples in the SAMM dataset, respectively. Functions in OpenCV and dlib libraries are used. Since the micro-expression recognition task focuses on certain parts of the face, regions outside the facial areas (hair, neck, background, ear, etc.) in the video frame may cause performance loss in expression recognition. A preprocessing task is performed to dispose of these noises’ areas. Thus, the irrelevant data for micro-expression recognition are excluded from the process. After detecting the face regions with the detector function in the dlib library, the trained model in the same library, which predicts the face region with 81 points, is used. The outer points are positioned to frame the outer part of the front view of the face, and the inner points are positioned as the mouth, eyes, eyebrows, and nose. The facial area was cropping and recorded after aligning the points on the temples of the forehead at the top and slightly above the chin at the bottom. After cropping face regions, all images' dimensions were resized to 224 x 224 for VGG16, MobileNetV2, EfficientNetB0, and 227x227 for AlexNet, and SqueezeNet.

In Figure 3, Face Detection, Face Mark, Face Alignment and Face Cropping operations are shown on an example of the SAMM dataset.

3.png

Figure 3. a) Face detection, face mark, b) Face alignment c) Face cropping operations

4.2 Optical flow

Optical flow is a method that can capture subtle movements between two frames and small changes in pixels according to the assumption of constant brightness [21, 33]. This method is used in areas such as action recognition, action detection, object tracking. Density-based Farneback [34] is used to detect the optical flow of each point. This method, which is also used in micro-expression studies [9, 35] was used in this study because of its many advantages. In addition to the advantage of fast computation, the Farneback optical flow technique, which has a pyramidal decomposition approach and produces relatively fewer errors in analyzing facial movements, is preferred [36]. Farneback optical flow estimates the approximate neighborhoods of each pixel using polynomial expansion with a quadratic polynomial. Mathematical equations of Farneback optical flow are given in Eqns. (2)-(6) [34].

$f_1(x)=x^T A_1 x+B_1^T x+c_1$        (2)

here, A is a symmetric matrix, b is a vector, and c₁ is a scalar. A new f₂ signal is generated using a global shift d.

$f_2(x)=f_1(x-d)$

$=(x-d)^T A_1(x-d)+b_1^T(x-d)+c_1$

$=x^T A_1 x+\left(b_1-2 A d\right)^T x+d^T A_1 d-b_1^T d+c_1$

$=x^T A_2 x+b_2^T x+c_2$            (3)

In Eq. (4), if the coefficients in quadratic polynomials are equalized, Eqns. (5) and (6) are obtained.

$A_1=A_2$      (4)

$b_2=b_1-2 A_1 d$      (5)

$d=-\frac{1}{2} A_1^{-1}\left(b_2-b_1\right)$       (6)

4.png

Figure 4. Optic flow processes with the onset and apex frame

In Figure 4, it is shown that the motion between the initial (onset) frame where the motion starts and the apex frame, where the expression is maximum, is obtained by optical flow.

The figure shown is an example of EP01_01f in Subject 19's in the CASME-II dataset, labeled happiness.

4.3 Data augmentation

Table 1 shows that negative emotions have more samples in all three data sets when compared to positive and surprise emotions. Class imbalance in the datasets makes classification in favor of the class with the large sample size of the SVM classifier [37]. The data augmentation technique was used due to the sampling imbalance in the micro-expression datasets used in this study. Thus, by creating a dataset that has balanced sampling, results in which the SVM to be used can classify more accurately are aimed. The classes in our dataset are 266 for negative sample, 112 for positive sample, and 88 for surprise sample. Data in other classes were increased by taking the negative class with the largest sample of data as a reference. At the end of the data increase, each class equals 266 optical flow image data. Data in classes with low samples are increased by rotating 90°, 180°, 270° degrees.

4.4 Convolutional neural network

1,000 deep features are obtained from fully connected layers of 5 different CNN models. These are the FC8 layer from VGG16, FC8 layer from AlexNet, Pool10 layer from SqueezeNet, Logits layer from MobileNetV2, and dense|MatMul layer from EfficientNetB0. 1,000 features taken from each layer are brought to the same plane using the MinMaxScalar method. This data is then combined and fed into the input of the PSO algorithm with combined 5,000 features. In Table 2, the models and the layers of the models from which the features are obtained are given.

Table 2. Selected layers of CNNs

4.5 Particle swarm optimization

In this study, the PSO optimization algorithm was used for feature selection. The parameters used in the experiment were respectively randomly adjusted in the range of Population Dimension (D) 1000, which gives the number of extracted features, Population Size (N) 10, which gives the number of populations, and Population Matrix (X) $0>x_i^t>1$. In cases where $x_i^t>0.5$, feature selection is made, otherwise feature selection is not made. When the Iteration of number 100 is taken, the number of function evaluations is N*100=1000, and the Number of KNN (k) is 5. PSO parameters are taken as 2 for Cognitive factor, 2 for Social factor, and 0.9 for Inertia weight. The fitness calculation equation is given in Eq. (7).

EroorRate $=1-$ ACC

SF (SelectedFeatures $)=\sum_{i=0}^D x_i=1$

Fitnes Function $=$ alpha $*$ ErrorRate $+$ beta*(S F / D)                  (7)

here, SF represents the total selected features, while the accuracy rate obtained in the K-NN classification of ACC selected features is taken as alpha=0.99, beta=0.01.

A total of 5,000 deep features obtained from 5 different CNNs are given to the selected optimization algorithm to choose the best features. According to the algorithm results, 2,746 different deep features of deep features were selected and provided as input data to the SVM classification.

4.6 Support vector machine

SVM has been used in research because it is found to be successful in statistical learning, modeling of data optimization, object detection as well as micro-expression recognition tasks [3, 5, 8]. In this study, linear, quadratic, finegaussian, and cubic kernel functions were used with SVM, and its superiority in the micro-expression recognition task was compared. The mathematical equations of the cores used in our study are given in Eqns. (8)-(11) [38].

Linear $\operatorname{SVM} k\left(x_i, x_j\right)=x_i^T x_j+c, c$ is a constant           (8)

Gaussian $k\left(x_i, x_j\right)=\exp \left(-\frac{\left\|x_i-x_j\right\|^2}{2 S^2}\right)$           (9)

Cubic $k\left(x_i, x_j\right)=\left(x_i^T x_j+1\right)^3$         (10)

Quadratic $k\left(x_i, x_j\right)=1-\frac{\left\|x_i-x_j\right\|^2}{\left\|x_i-x_j\right\|^2+c}$       (11)

here, X_i and X_j are input vectors, k () is kernel function. S is the bandwidth parameter, which determines how quickly the similarity metric drops as the samples move away from each other.

4.7 Performance metrics

In machine learning, other measures besides accuracy are used to evaluate the success of the model on the data set and to analyze the data in more detail [39-41]. The accuracy value alone gives only how many of the predictions made are correct. Therefore, in this study, the confusion matrix is used to analyze the classification performance of the models used. The parameters used in the complexity matrix are True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN). The mathematical equations of these measurements are given in Eqns. (12)-(17) [42].

$\operatorname{Accuracy}($ Acc $)=\frac{T P+T N}{T P+F P+T N+F N}$          (12)

Sensitivity $(S n)=\frac{T P}{T P+F N}$      (13)

$\operatorname{Specificity}(S p)=\frac{T N}{F P+T N}$        (14)

Recall $=\frac{T P}{T P+F N}$       (15)

Precision $=\frac{T P}{T P+F P}$      (16)

$F_{\text {score }}=\frac{2 * \text { Precision } * \text { Recall }}{\text { Precision }+\text { Recall }}$       (17)

In addition, model performance was evaluated with a ten-fold cross-validation method. In the cross-validation method, the data set is divided into ten equal parts that nine parts are training data rest one part is test data. This situation continues until each part has been test-data once.

Nowadays, classification of ME images has an important place in many fields such as forensic informatics, security, and education. In our study, it was observed that the F1 scores of accuracy rates could not exceed 72% in the classification made using CNN models on the combined data set. To increase the success, it has been observed that the success rates increase in the classification made by duplicating the data set with the data augmentation technique. When the increasing rates are examined, While the accuracy rate exceeding 80% and the F1 measurement rate exceeding 81% were obtained. Table 3 that values up to 90% were obtained in other measurements. From this, it can be concluded that the data augmentation technique will positively affect many CNN models in increasing the ME recognition performance. In the classification using four different SVM kernels in Table 4, made with feature maps taken from the last connected layers of CNN models, the experimental results could not exceed the performance when Table 3 was taken as the basis. The results obtained by combining the feature maps in Table 5, on the other hand, partially step up onto the performance. It can be concluded that there is a need for more besides different classification techniques. It can be concluded that the reason for this is that more than different classification techniques are needed. As a matter of fact, in the next step, the results obtained from quadratic and cubic kernels with the experiment using the PSO algorithm for the selection of the best features caused a noticeable increase in Table 6. In the accuracy and F1 measurements, which reached the highest values, the values of 0.8243 and 0.8232 were obtained, respectively. In this study, the highest performance measurements were obtained among the experiments carried out so far. Thus, it has been observed that the data obtained by feature selection has a positive effect on increasing the ME recognition performance. In addition, Table 7 shows that the experiment using the PSO algorithm and cross-validation technique with SVM kernels there were significant improvements in classification performance for all measurement values compared to the initial values. More concretely, compared to the best measurement values in the composite data set in Table 3, the performance increase was 16.41% for accuracy, 21.58% for sensitivity, 11.32% for specificity, 18.28% for precision, and 20.26% for F1. Confusion matrices and AUC-ROC graphs of methods and techniques are given to analyze ME classification performance in depth. The best results obtained in our experiments are shown in Table 8.

In addition, a comparison of our work with the latest technology models in the ME field is presented in Table 9. The results show that our model is competitive and satisfactory.

In the study [15], the success of different optical flow methods in ME recognition is presented in Table 3. Recognition results produced using the Farneback method are given as the two best methods that outperform each other in different block sizes, together with TVL1. In addition, it has been shown as another advantage for the Faneback method, which is faster than the others in calculating the analysis of facial movements [36]. It should be considered that modified Pso derivatives and different SVM kernels can give more positive results in increasing ME recognition accuracy.