Depression Micro-Expression Recognition Technology Based on Multimodal Knowledge Graphs

According to the World Health Organization (WHO), approximately 350 million people worldwide suffer from depression [1], with a growth rate of about 18% over the past decade. Depression has become the second leading cause of death among humans. WHO predicts that by 2030, depression will be the leading cause of global disease burden [2]. Additionally, depression can lead to suicide, with an estimated 800,000 suicides occurring annually worldwide, up to 50% of which occur during depressive episodes. The likelihood of suicide among individuals with depression is nearly 20 times higher than that of the general population. According to authoritative surveys [3], 24% of Chinese college students suffer from depression, with the rate rising to 35% in prestigious universities. These surveys reflect a high prevalence of depressive states among college students, with 41% of those affected having taken a leave of absence due to the illness, and some having committed suicide as a result of depression, which warrants attention. Depression among Chinese individuals tends to manifest physically, with many patients seeking psychiatric help primarily for somatic symptoms rather than psychological discomfort. The somatization reporting rate among Chinese people is much higher than that of Western populations, with more than 70% of patients presenting with somatic symptoms as the primary complaint. This has resulted in a low clinical diagnosis rate for depression in China, meaning the actual number of depression patients in China far exceeds the reported figures.

Early detection of signs of depression is crucial for timely treatment and recovery. Traditional diagnostic methods for depression rely on patient self-reports and psychological assessment scales. However, some patients report somatic symptoms, some choose to hide their emotions (such as patients with smiling depression), and some deliberately skew their answers toward "positive" options when filling out psychological assessment scales, leading to many limitations in traditional diagnostic methods. With the development of emotion recognition technology, collecting multimodal indicators such as physiological and behavioral data, combined with statistical analysis, machine learning, and deep learning techniques, to identify features that differentiate depression patients from healthy individuals has become a research focus for many scholars. Common physiological signals include EEG [4], MRI [5], and neuroimaging [6]; common behavioral signals include eye movement [7], gait [8], speech [9], and facial expressions [10]. Compared to other physiological and behavioral signals, facial expressions have advantages such as being non-intrusive, easy to obtain, and low-cost in the collection process. Facial expressions can be categorized into macro-expressions and micro-expressions. Macro-expressions can be easily identified in daily life, while micro-expressions are fleeting facial expressions that reveal a person’s true feelings and emotions. Studies have shown that micro-expression recognition may have significant predictive value for depression and holds considerable clinical value. Micro-expressions are subtle and short-lived, making it difficult to identify their types with the naked eye, and studies have shown that manual recognition accuracy is only 47%. Currently, deep learning-based micro-expression recognition methods have become a research focus. For example, Islam MD. Islam et al. [1] utilized deep learning’s multi-channel feature fusion and ensemble classification techniques to recognize emotional information in facial micro-expressions, and Durga et al. [2] designed an adaptive Tiefes FCNN model based on deep learning for micro-expression recognition. These studies have demonstrated that deep learning techniques outperform traditional machine learning methods in micro-expression recognition. However, most micro-expression recognition studies are based on facial image features, with room for improvement in accuracy and precision. According to Mehrabian’s rule, emotional communication consists of 55% facial expressions, 38% tone of voice and body movements, and 7% language. Thus, in depression recognition research, combining facial expression and speech data analysis for modeling can provide more emotional information than facial expression analysis alone [11, 12].

This study first analyzes the association between micro-expression facial action units (AUs) and depression, collecting multimodal data from conversations between students and psychological counselors, including speech, video, and psychological scale data, to establish a multimodal psychological dataset for college students. A micro-expression recognition system is then developed based on deep learning techniques, using image frames from videos to identify and judge students' micro-expressions, revealing their psychological changes. Simultaneously, speech is converted into corresponding feature text, and combined with psychological scale data, a multimodal micro-expression knowledge graph is constructed to analyze each feature and generate the final target emotional text, assessing whether there is a risk of depression.

3.1 Pre-training database selection

For micro-expression research, the publicly available micro-expression databases include the CASME database, CASMEII database, SAMM database, CAS(ME)² database, CAS(ME)³ database, and SMIC database. Among these, the CAS(ME)³ database [14] provides approximately 80 hours of video, including 1,030 manually annotated micro-expressions and 3,364 macro-expressions. Such a large sample size allows for effective validation of intelligent micro-expression analysis methods while avoiding database biases. Additionally, this database is the first to provide depth information as an additional modality and includes over 13,000 seconds of physiological and speech signals, which facilitates multimodal micro-expression analysis. Therefore, this study uses this database to train deep learning neural networks.

In the field of deep learning, transfer learning is an important technique that can help build high-performance neural networks on limited datasets. The core idea of transfer learning is to use a model already trained on the CAS(ME)³ database, apply weight transfer, and then apply it to the collected multimodal psychological dataset of college students, aiming to reduce the need for training data and improve the performance of deep learning models. Transfer learning can be divided into two main stages: pre-training and fine-tuning. In the pre-training stage, a neural network model was trained using the CAS(ME)³ dataset, which is commonly referred to as the pre-trained model. In the fine-tuning stage, the pre-trained model was applied to a new task—college students' depression recognition—and fine-tuned using the data from the new task. Through this method, we can build a high-performance neural network on the CAS(ME)³ dataset and then fine-tune the network parameters using transfer learning techniques for multimodal psychological recognition tests of the students in our school.

3.2 Multimodal psychological data collection of college students

We collected the weekly routine interview video records of 100 freshmen with abnormal results in the school psychological survey during their first semester at the school’s psychological counseling room. The interview environment had normal classroom lighting, with illumination provided by indoor fluorescent lights and natural light from windows. Recording equipment included a high-speed camera and a Microsoft Kinect V2 camera. The high-speed camera was used to capture the students' facial micro-expressions, while the Microsoft Kinect V2 camera collected the speech content and high-definition RGB camera images during the conversations. Each student’s conversation video lasted about 15-30 minutes. During the data collection process, participants sat in a well-lit room, with the camera fixed approximately 0.5 meters in front of them to ensure the entire facial region was captured. The high-speed camera was set to continuously capture facial images at 120 fps, ensuring sufficient lighting and no obstructions in front of the subject during video recording. Before data collection, the research team obtained informed consent from all students, explaining the purpose of the research. To protect participants' privacy, the collected data was anonymized and access was restricted, allowing only research team members involved in the study to handle and analyze the data. Key frames containing micro-expressions were selected from the high-frame-rate image sequences and stored to form a dataset rich in micro-expressions. Ultimately, we collected multimodal data from conversations between students and psychological counselors, including speech data, video data, and psychological scale data, and recorded the final diagnostic results from the psychological counselors for later comparison and validation against the computer recognition results.

5.1 Image feature extraction using Gabor wavelet transform

We used a high-speed camera to capture real-time images of the face and complete the capture of facial micro-expressions. After filtering, denoising, and grayscale equalization processing of the images, Gabor wavelets were used for feature extraction [16]. Gabor wavelets are sensitive to image edges and provide good directional and scale selection characteristics. They are also insensitive to lighting changes, offering good adaptability to variations in illumination. The two-dimensional Gabor function is similar to enhancing low-level image features such as edges, peaks, valleys, and ridge contours, effectively enhancing information about facial key components such as the eyes, nose, and mouth, making it well-suited for micro-expression feature extraction. As described in Table 1, the identification of micro-expression AUs is mainly concentrated in the cheeks, eyes, eyebrows, nose, and lips regions. Therefore, in expression recognition, we also focus on extracting facial features from these five regions.

The Gabor function is a complex sinusoidal function modulated by a Gaussian function, capable of extracting local frequency domain features within a given region. A typical 2-D Gabor function h(x,y) and its Fourier transform h(u,v) have the following forms:

$\left\{\begin{array}{l}g(x, y)=\frac{1}{2 \pi \sigma_x \sigma_y} \exp \left(-\frac{1}{2}\left(\frac{x^2}{\sigma_x^2}+\frac{y^2}{\sigma_y^2}\right)\right) \\ h(x, y)=g(x, y) \cdot \exp (2 \pi j W x) \\ H(x, y)=\exp \left(-\frac{1}{2}\left(\frac{(u-W)^2}{\sigma_u^2}+\frac{v^2}{\sigma_v^2}\right)\right)\end{array}\right.$               (4)

where, g(x,y) is the Gaussian function used for modulation; σ_x and σ_y are the standard deviations on the two coordinate axes, determining the size of the filter's effective area; and W is the frequency of the complex sinusoidal function on the horizontal axis.

The Gabor feature map is obtained by convolving the cropped and preprocessed facial images with the sum function of Gabor wavelets, with the calculation method shown in formula (5).

${{P}_{u,v}}(z)=I(z)*{{G}_{u,v}}(z)$               (5)

In the formula, z=(x,y) represents the coordinates of the pixel points in the image, P_u,v(z) represents the feature image output after the convolution operation, I(z) represents the facial image to be processed, G_u,v(z) represents the two-dimensional Gabor wavelet kernel function, and * denotes the convolution operation.

The expression for the Gabor wavelet kernel function is given by formula (6):

${{G}_{u,v}}(z)=\frac{k_{u,v}^{2}}{{{\sigma }^{2}}}\exp (-\frac{k_{u,v}^{2}}{2{{\sigma }^{2}}})[\exp (i{{k}_{u,v}}z)-\exp (-\frac{{{\sigma }^{2}}}{2})]$               (6)

Its real and imaginary parts are given by formulas (7) and (8):

$\operatorname{Re}({{G}_{u,v}}(z))=\frac{k_{u,v}^{2}}{{{\sigma }^{2}}}\exp (-\frac{k_{u,v}^{2}}{2{{\sigma }^{2}}})[\cos ({{k}_{u,v}}z)-\exp (-\frac{{{\sigma }^{2}}}{2})]$               (7)

$\operatorname{Im}({{G}_{u,v}}(z))=\frac{k_{u,v}^{2}}{{{\sigma }^{2}}}\exp (-\frac{k_{u,v}^{2}}{2{{\sigma }^{2}}})[\sin ({{k}_{u,v}}z)]$               (8)

where, $k_{u, v}=\binom{\kappa_v \cos \varphi_u}{k_v \sin \varphi_u}$ is the center frequency of the filter; $k_v=2^{\frac{v+2}{2} \pi}, \varphi_u=u \frac{\pi}{k} ; u(u=0,1,2, \ldots, 7)$ is the orientation of the Gabor filter; $v(v=0,1,2,3,4)$ is the scale; $k$ is the total number of orientations; $i$ is the negative operator; and $\sigma=2 \pi$. Gabor feature extraction includes the following operations: modulus extraction, feature dimensionality reduction, feature normalization, and feature selection.

(1) Modulus Extraction: Based on the real and imaginary parts returned by formulas (7) and (8), the modulus is used as the image feature.

$\bmod =\sqrt{{{\operatorname{Re}}^{2}}+{{\operatorname{Im}}^{2}}}$               (9)

(2) Feature Dimensionality Reduction: The feature dimensions extracted by Gabor wavelet transform are generally high. As shown in formulas (7) and (8), we use Gabor filters with 8 orientations and 5 scales, resulting in 40 transformed images; this leads to a high feature dimension (concatenation of 40 image features). Therefore, it is necessary to perform feature dimensionality reduction on the modulus images. Here, downsampling is applied to each modulus image using the resize function in OpenCV: cv2.resize(img, (0,0), fx=1/4, fy=1/4).

(3) Feature Normalization:

The purpose of normalization is to constrain the preprocessed feature data within a certain range. In this study, the Z-score standardization method is used, and the normalization formula is as follows:

$x=\frac{x-\mu }{\sigma }$               (10)

where, μ represents the mean of all sample data, and σ represents the standard deviation of all sample data.

(4) Feature Selection:

After the above three steps, the features of each modulus image can be concatenated as the feature vector of the input image. However, the obtained feature vector still contains a large amount of redundancy. Therefore, Principal Component Analysis (PCA) is used for feature dimensionality reduction [17] to alleviate the dimensionality problem, build an effective data model, and reduce the time consumption for recognition, thereby extracting more representative image feature information.

5.2 Micro-expression recognition model based on deep learning

In recent years, CNNs have shown outstanding performance in facial expression recognition tasks. The main advantage of CNNs is that they reduce the number of parameters that need to be trained through parameter sharing and sparse connections, giving them an edge in handling large-scale data. Additionally, their transfer learning capabilities allow model parameters trained in other domains to be transferred to facial expression recognition tasks, thus accelerating the training process and improving model performance. However, since CNNs can only efficiently process regular Euclidean data such as grids and sequences, they cannot effectively handle graph-structured non-Euclidean data like social media networks, chemical structure data, biological protein data, and knowledge graph data. The structure of graphs is typically highly irregular. In micro-expression recognition tasks, CNNs only analyze the key facial AUs in micro-expression image sequences and cannot closely associate facial AU combinations with corresponding emotional labels. GCNs [18], which are the application of CNNs on graph-structured non-Euclidean data, address this limitation. Since the facial texture features extracted from AUs cannot directly reflect emotional categories, they need to be mapped as intermediate features into different category spaces. The AUs from different facial regions are independent yet interconnected; therefore, GCNs can be used to analyze the dynamic connections between different AUs, obtaining deep spatiotemporal features and exploring the dependencies between key facial regions, leading to more accurate micro-expression classification. We divide the Gabor wavelet transform-extracted feature dataset into training, testing, and validation sets and construct a shallow CNN model with convolutional layers, pooling layers, fully connected layers, and dropout layers as the core. Considering that it will later be combined with GCNs to build a deep network model, as the number of layers increases, the training of the network will become increasingly difficult. To address this issue, the shallow CNN is further improved by introducing the concept of residual networks. Suppose that in a single training module, the fused feature input of the original shallow network model is x. After multi-layer complex mapping calculations cascaded by hidden layers in the network, the output is the deep feature H(x) of the micro-expression. Since the complexity of the network model obtained by connecting multiple modules is high and difficult to train, the idea of residual structure is adopted, using shortcut connections to directly input the shallow feature xxx into the output of the training module as the initial residual. The deep feature H(x) in this training module can be expressed as:

$H(x)=F(x)+x$               (11)

If F(x) is close to 0, it constitutes an identity mapping $H(x) \approx x$, which can further simplify the complex model. The deep feature training network is decomposed into multiple shallow multi-scale training modules based on residual structures. The CNN model with added residual modules is shown in Figure 1.

1.png

Figure 1. CNN model with residual modules

2.png

Figure 2. CNN+GCN-based micro-expression recognition network framework

Based on the CNN model, this paper improves and constructs a micro-expression recognition model based on a CNN+GCN transfer learning network. It utilizes the unique advantages of GCN to obtain node attributes and hidden association information between nodes in the emotional label vectors, helping to capture the dependency between image data and corresponding emotional labels in micro-expression sequences. The overall network framework can be divided into two parts: the CNN-based image feature training module and the GCN-based emotion classification module. The schematic diagram is shown in Figure 2. First, the dataset images undergo preprocessing, Gabor wavelet transform, and PCA dimensionality reduction. Then, the feature dimensions of the images are reduced to D through global max-pooling in CNN, and the output C represents the number of emotion categories in the dataset. In the GCN-based emotion classification module, different category label word embedding vectors are obtained according to the combinations of facial AUs in key regions. GC represents performing graph convolution operations on each label vector. The original label word embedding vectors have a dimension of d, which is transformed to dimension D through feature mapping, and this dimension is determined by the image feature representation dimension obtained from CNN.

To save training time, improve the performance of the neural network, and avoid overfitting due to small sample sizes, we will next implement transfer learning of the network model. First, in the structure of Figure 2, the DATASET uses a sufficient sample size from the CAS(ME)³ micro-expression dataset to identify emotion categories and pre-train the model. The image features learned by CNN are max-pooled to obtain new global features x:

$x=G[{{f}_{MP}}({{F}_{CNN}}({{I}_{c}},{{w}_{CNN}}))]$               (12)

where, I_c represents the feature representation of the input image; w_CNN represents the weight parameters in the CNN network model; F_CNN denotes the iterative operations of the hidden layers in the CNN model; f_MP represents the max-pooling function; and G is the global computation function of the pooling features.

By associating different AUs with corresponding emotion categories, GCN is used to train the target classifier, obtaining hidden association information between different category labels. The feature of the preceding layer's nodes in the current network is repeatedly iterated as the input of this layer, updating the node features to realize the strategy modeling of the target classifier on label vectors, thus classifying micro-expression images.

3.png

Figure 3. Transfer learning network recognition model

After the pre-training is completed, the "knowledge" (network weights) obtained during the training with the model in Figure 2 is transferred to the target model, and then parameter fine-tuning is performed in the recognition task of the target model. The transfer learning network recognition model is shown in Figure 3.

7.1 Deep learning model experimental results

Here, we compare the micro-expression recognition results on the CAS(ME)³ dataset using the original CNN model proposed in Section 5.2, the CNN model based on residual modules (here abbreviated as Res+CNN), and the CNN+GCN network model. The network structure of the CNN+GCN model mainly consists of graph convolution units and a CNN based on residual modules. By cascading the graph convolution units with the residual modules, the CNN based on residual modules captures the intrinsic relationships of micro-expression features, forming new feature vectors. The graph convolution units are used to map the label graph data to the feature learning space of the CNN, capturing hidden associations between micro-expression facial features and corresponding emotional labels. Table 2 shows the main parameter configurations of the three comparison models. In the CNN model and the CNN model based on residual modules, the convolution units consist of 2 convolution layers with an initial number of 32 kernels and a kernel size of 3*3. In the CNN+GCN model, the number of convolution kernels is changed to 64, the initial value of m is set to 1, and for each convolution unit, the value of m is multiplied by 2, doubling the number of convolution kernels. During training, the Adam optimizer is used to automatically adjust the learning rates of various parameters, with the batch size set to 64; initial learning rate set to 0.001; learning decay rate set to le-5; momentum set to 0.9; and iterative cycle set to 100. Adaptive supervision is used to adjust the learning rate. Each convolution layer adds BN regularization, and the EarlyStopping callback function is used to interrupt training when the model performance no longer improves, alleviating overfitting and enhancing the model's generalization ability. Figure 5 shows the recognition accuracy comparisons of the three different models on the CAS(ME)³ dataset.

Table 2. Main parameter configurations of different network models

5.1.png

(a) Original CNN model accuracy

5.2.png

(b) Res+CNN model accuracy

5.3.png

(c) CNN+GCN model accuracy

Figure 5. Recognition accuracy comparison of three different models on the CAS(ME)³ dataset

6.png

Figure 6. Accuracy curve of CNN+GCN transfer learning network micro-expression recognition model

To address the problem of potential model overfitting due to insufficient student micro-expression data samples collected in this study, we use the CAS(ME)³ dataset micro-expression recognition model based on the CNN+GCN network. The feature images obtained from the CAS(ME)³ dataset are used as the training and test sets, expanding the sets in a 4:1 ratio, and pre-training is conducted on the student micro-expression dataset collected in this study. The weight-based transfer learning network recognition model from Figure 3 is used to recognize micro-expressions, and the results are shown in Figure 6.

Figure 6 shows that the CNN+GCN model achieves a maximum recognition rate of 71.46% for micro-expression features on the CAS(ME)³ dataset. The weight-based transfer learning network recognition model alleviates the impact of the small number of student micro-expression sample sequences on network training. During the continuous training and optimization process of the network, the model achieves a maximum recognition rate of 77.14%.

7.2 Comparison of single-modality and multi-modality evaluation results

Here, we compare the recognition results of three single modalities: micro-expression recognition based on speech data, emotion recognition based on scale detection, and micro-expression recognition based on video images, with the recognition results of the multimodal fusion model based on "speech + psychological scale + video image" proposed in this paper for depression. The experimental evaluation indicators use the average accuracy of four emotions, and the results are shown in Table 3. As shown in Table 3, the proposed multimodal fusion model achieves higher accuracy than the three single-modal models, with an average accuracy of 80.13%.

Table 3. Performance comparison of multimodal fusion model and single-modal models