Enhanced Autism Spectrum Disorder Facial Expression Recognition Using Hybrid Weighed Quantum Particle Swarm Optimization with Fast Mask Recurrent Convolutional Neural Network

ASD is a neurological illness that impacts behavior and communication. Although it can manifest at any age, symptoms typically appear in a child's early years. Many individuals with ASD exhibit irritability, sleep problems, and a lack of desire to interact with others, maintaining a steady demeanor [1]. ASD symptoms can vary greatly between individuals. Severe autism often results in difficulties with verbal communication or being nonverbal altogether, making it hard to identify and express feelings, and complicating everyday tasks. People with ASD frequently struggle with self-regulation, emotional interpretation, and understanding and labeling others' emotions. These differences can lead to social withdrawal, especially when others react negatively to the atypical responses of individuals with ASD [2].

Research indicates a lack of studies utilizing static or dynamic facial stimuli displaying real, spontaneous emotions. Staged emotions are easier to detect than spontaneous ones, but the specific emotion being evaluated can affect the accuracy of Facial Emotion Recognition (FER). Staged expressions are more stereotypical, whereas real displays, such as grief, vary greatly [3]. Researchers should pay more attention to the distinction between staged and authentic emotions in ASD studies. Extensive research has highlighted notable differences between ASD and control groups in interpreting staged facial displays of emotion. New studies compare autistic individuals to controls in creating staged emotional expressions, an area of study known as computational psychiatry [4]. This field seeks to bridge the gap between scientific findings and psychiatric symptoms by studying brain information processing. Consequently, computational work aims to understand the mechanism of facial emotion identification and its impairment in ASD [5].

ASD is a lifelong condition manifesting in early childhood, affecting behavior, social interactions, communication, and learning. The spectrum label reflects the wide variety of symptoms, including varying degrees of IQ, social interaction, verbal communication, self-stimulatory activities, sensory abilities, and motor skills [6]. Symptoms can vary widely; for instance, a person may have a high IQ but little interest in social interaction, be overly sensitive to noise but underreact to pain, excel with gross motor skills but struggle with fine motor tasks [7]. People with ASD may have trouble expressing themselves verbally, avoid eye contact, and exhibit repetitive behaviors like organizing objects or repeating phrases. They often seem isolated, living in a bubble [8].

ASD children struggle to express their feelings, hindering early development as parents and caregivers find it difficult to understand their behavior. Studies show a strong correlation between psychological reactions and physiological changes, with emotional responses identified through changes in physiological markers. Consequently, important features are extracted from recorded physiological signals to better understand these reactions [9].

1.1 Problem statement

Successfully identifying and understanding facial expressions might be difficult for those with ASD. Their ability to recognize facial expressions can have a big influence on their ability to interpret emotions, interact socially, and feel good overall. The unique demands of people with ASD may not be adequately met by current facial expression detection technology, which might hinder their capacity to interact with others and function in social situations.

As a result, it is imperative to create improved recognition methods designed especially for people with ASD. These methods ought to increase facial expression recognition precision while also taking into consideration the special traits and difficulties connected to ASD. By solving this issue, researchers can enable people with ASD to interpret and react to social cues more effectively, which will improve their quality of life and integration into society [10].

1.2 Motivation

Improving the identification of abnormal facial expressions in persons with ASD is driven by the significant influence it may have on their lives. via strengthening their capacity to precisely read and comprehend facial emotions. Interpersonal communication and interaction difficulties are common for people with ASD, which can cause feelings of loneliness and make it challenging for them to build connections. Through tackling the problem of modified facial expression identification, researchers hope to give those useful resources and assistance to better navigate social settings. By utilizing these methods, it will be possible to develop effective and customized facial expression detection systems that address the particular traits of ASD. Our ultimate driving force is the possibility of significantly improving the lives of people with ASD by giving them the tools they need to communicate with themselves, interact with people more effectively, and engage fully with the community.

The proposed study aims to close a sizable gap in existing approaches to the identification of changed facial expressions in people with ASD diagnoses. The creation of precise identification systems catered to individuals with ASD is hampered by the inability of current methods to capture the subtleties of facial expressions unique to this demographic. The article presents an original structure that brings together two state-of-the-art methods to close this gap: The overall objective of the research is to combine many methodologies in a complementary way to provide an improved way of detecting individuals with ASD based on their altered facial expressions. An innovative optimization technique that combines ideas from particle swarm optimization and quantum computing is the hybrid WQPSO–FMRCNN method. This method can successfully explore complicated solution spaces because it uses weighted classical operations to direct the optimization procedure.

1.png

Figure 1. The proposed architecture

Figure 1 illustrates the hybrid WQPSO-FMRCNN structure to analyze sequential information, including facial movements taken photos, or videos. Understanding the special difficulties people with ASD have reading and reacting to their facial reactions is essential to the study project. A vital component of interpersonal interaction and psychological understanding—areas where people with ASD frequently struggle—is altered facial expression interpretation. To overcome these obstacles, a hybrid solution has been suggested that particularly customizes the identification system to meet the demands of people with ASD. The project intends to empower people with ASD to manage social interactions with greater trust and knowledge, eventually enhancing their quality of life, by creating a more efficient and accessible tool. The study suggests a unique approach to improve the identification of abnormal facial expressions in ASD patients by combining Hybrid WQPSO – FNRCNN. By combining these cutting-edge methods with an emphasis on the distinctive features of ASD, we may help people with ASD comprehend and react to their facial reactions more effectively, which promotes greater interpersonal relationships and interpersonal interaction. The overall layout of the suggested model is shown in Figure 1. The model is then trained using the FMRCNN once the hyper parameters have been optimized. Optimizing the CNN hyper parameters and then training a model are the two main components of the proposed WQPSO-FMRCNN Technique.

3.1 Data set

This study's feature extraction for emotion recognition relies on physiological signals culled from the IAPS database. To create a database of 4D data, we had 14 individuals execute the 6 most fundamental facial expressions. Participants' ages, ethnicities, and the kinds of facial characteristics they exhibited (such as hair, spectacles, and head coverings) were all carefully considered to ensure that the data set was representative of the population at large. Everyone who took part had many sessions recorded. The data has been organized into seven categories: the six most common facial expressions and one for neutral emotions. "Subset 1" in the results section denotes the situation when the same group of people served as the subjects for both the training and testing sets of frames. Regarding "subset 2," nearly every individual from the test set was absent from the training set.

3.2 Image pre-processing

It is a crucial step in facial expression recognition systems, particularly in the context of recognizing altered facial expressions in individuals with ASD. To enhance the efficiency of later processing steps, this procedure combines several approaches to improve the quality of the input pictures and extract pertinent information. Typical methods for preparing images include the following:

Image Resizing: Ensuring consistent treatment by resizing images to a common format will help guarantee consistency in input dimensions, across various samples.

1. Normalization: Images are better suited for determining features when pixel values are normalized to a common scale, like [0, 1] or [-1, 1]. This helps reduce fluctuations in brightness and contrast.

2. Histogram Equalization: To increase picture contrast and make facial characteristics and gestures more visible, the histogram equalization process redistributes the intensity of pixels.

3. Face Recognition and Realignment: By concentrating analysis on pertinent areas, recognizing faces, and alignments within pictures can increase reliability before identification.

4. Feature Extraction: Extracting relevant features from facial regions, such as key points or landmarks, using techniques like facial landmark detection or Local Binary Pattern (LBP) descriptors.

By applying these pre-processing techniques, the input images are transformed into a standardized and optimized format, which can enhance the performance and accuracy of subsequent stages, such as feature extraction and recognition, in the facial expression recognition system tailored for individuals with ASD shown in Figure 2.

2.png

Figure 2. Facial recognition using the proposed system

3.3 Feature selection using modified trapezoidal fuzzy membership genetic algorithm

A search heuristic that attempts to capture the essence of evolution is known as a Genetic Algorithm (GA). Optimization and search issues frequently benefit from this heuristic's application. Among the broader category of Evolutionary Algorithms (EA), genetic algorithms are a subset that finds optimal solutions to optimization problems by mimicking natural selection, mutation, inheritance, and crossover.

GA work by allowing a population of strings—the genome's genotype or chromosomes—to develop towards better solutions to optimisation problems. There are other alternative encodings for solutions, however the most common is binary, which is a string of 0s and 1s. In most cases, evolution occurs in generations beginning with a population of randomly created individuals. At the beginning of each generation, all of the individuals in the population are assessed for their fitness. Then, a number of people are randomly chosen from the current population and changed, either by combining them or by randomly altering their genes, to create a new population. The programme iteratively uses the new population in subsequent iterations. The algorithm may or may not have produced a good solution if it ended due to an excessive number of generations.

For the purpose of automatically representing physiological signal features using attribute value ranges [0-1], the Trapezoidal Fuzzy membership function using GA is shown in Figure 3. A gene expression matrix x and four scalar parameters a, b, c, and d determine the trapezoidal curve, which is expressed as:

$f(x)=\left\{\begin{array}{lr}0, & k \leq a \\ \frac{k-a}{b-1}, & a \leq i \leq b \\ 1, & b \leq i \leq c \\ \frac{d-i}{d-c}, & c \leq i \leq d \\ 0, & d \leq k\end{array}\right.$   (1)

$fitness_x= \begin{cases}\frac{1}{\left(1+f i t_x \cdot w_x\right)} & { if } f_x \cdot w_x \geq 0 \\ \frac{1}{\left(a b s\left({ fit}_x \cdot w_x\right)\right)} & { if } f_x \cdot w_x<0\end{cases}$    (2)

3.png

Figure 3. Feature selection using modified trapezoidal fuzzy membership genetic algorithm

3.4 Emotion recognition

We are able to read other people's emotions and moods from their facial expressions, which in turn helps us to adapt our actions and responses. Thus, it is widely believed that the capacity to read and understand socially important cues from facial expressions is a prerequisite for most forms of reciprocal communication and social engagement. In most cases, understanding how autistic children feel is challenging. The majority of works that discuss emotions state that they are inherently complicated.

·Variety of emotion elicitation methods.

·A technique for representing emotional states using either dimensional.

·Methods for selecting a reference emotional state based on annotated stimuli or subjective judgements.

·Assessment technique for estimators.

A database called Karolinska Directed Emotional Faces is used to train and evaluate our emotion recognition algorithms in this study. A total of 70 people, including 35 men and 35 women, ranging in age from 20 to 30 years old, are included in the corpus. From five different perspectives, each person exhibits seven distinct emotional expressions shown in Figure 4. Using a grid that centred the subject’s eyes and lips in predetermined picture coordinates, all of the photos were shot in a controlled setting.

4.png

Figure 4. Six emotions

5.png

Figure 5. CK+ dataset 5 emotions: happy, digest, angry, fear and sad

After locating the face, we removed any extraneous spatial characteristics, such as the backdrop to create a feature vector. To expedite the training process, the face photos were grayscaled and reduced in size to a typical 120×110 for the Gabor filter and 100×100 for the CNN. The KDEF database provided the sample face pictures shown in Figure 4. Another well-known dataset for emotion identification and face detection, the Extended CohnKanade is used to test the WQPSO-FMRCNN model on unseen data. A total of 118 individuals contributed 327 sequences to this collection. Since it includes the greatest information relating to emotions, the peak frame of each sequence is the only one used in this work. Figure 5 shows the results of applying the identical pre-processing to this dataset as to the KDEF dataset. By running our hybrid model on the CK+ dataset, it stacks up against the work of another study that employs a WQPSO-FMRCNN combination.

3.5 Hybrid proposed method

The WQPSO-FMRCNN takes its cues from the original PSO which employed a quaternion space for the encoding of virtual swarm (physiological signal characteristics) instead of a Euclidean. This would allow the search to focus on the most fruitful regions of the search space. By modeling physiological signal properties using quaternions, the WQPSO-FMRCNN departs from the original PSO. Instead, the quaternion algebra is used on this quaternion to capture the characteristics of physiological signals. Quaternions are formal expressions that fulfill Hamilton's equations.

$x y=z, y z=x, z x=y$ (3)

$xy=-z,zy=-x,xz=-y$  (4)

${{x}^{2}}={{y}^{2}}={{z}^{2}}=-1$ (5)

Adding and subtracting quaternions, and performing scalar multiplication and multiplication, are all defined by the quaternion algebra. Here add two unary functions to pure quaternion algebra.

Quaternion defined in Eq. (6):

$qrand()=\left[i_x=N(0,1) \mid\right.$ for $\left.x=0, \ldots, 3\right]$    (6)

where, N (0,1) - random number in the Gaussian distribution. Each component is initialized,

$qzero ()$=$\left[i_x=0 \mid\right.$ for $\left.x=0, \ldots, 3\right]$      (7)

Using the qrand() function, initialises the quaternion population. Using the norm function, extract the solution $p s f s=\left(p s f s_0, p s f s_1, \ldots p s f s_D\right)$ from the Dth quaternion's vector qi in the following way:

$p s f s_y=\left|\mid q_{x y}\right\|$, for $y=1$ to $D$          (8)

$r_{x y}^2=d_{x y}\left(q_{x y}\right)=\sqrt{\left(w_x-w_y\right)^2}$    (9)

where, $q_{x y}$ - xth virtual swarm position; $q_{y}$ - yth virtual swarm position in the search space; Swarm moving x to another more attractive swarm x to more attractive swarm y is expressed as follows:

$d_{x y}\left(q_{x y}\right)=\sqrt{\left(w_x-w_y\right)^2}$  (10)

$q_x=q_x+\beta_0 e^{-\gamma r_{x y}^2}\left(q_y-q_x\right)+\alpha \cdot \epsilon . qrand()$       (11)

3.6 Automatic feature extraction with autoencoder

The proposed WQPSO-FMRCNN approach incorporates an autoencoder for feature extraction and feature selection shown in Figure 6 to enhance its performance in effectively classifying the input picture.

6.png

Figure 6. The proposed WQPSO-FMRCNN framework using auto encoders

An unsupervised neural network called an autoencoder tries to compress the input data as much as possible. The autoencoder's design limits the architecture to a single point where it can recover the picture. When a nonlinear connection can be used to represent the relationship between the independent and dependent datasets, it is used to minimize the dataset's dimensions. The encoder, bottleneck, and decoder are the three primary components of an autoencoder, as seen in Figure 6. The encoder begins by prioritizing features to include in the input data. The decoder, meantime, makes use of the critical characteristic to restore the original inputs. By preserving the characteristics needed for data reconstruction, autoencoders lower the data dimension shown in Figure 7. There is some loss in the output of autoencoders, but otherwise, it is the same as the inputs.

7.png

Figure 7. The five primary used emotions

$Y=f\left( x \right)={{S}_{F}}\left( WX+{{b}_{x}} \right)$  (12)

where, ${{S}_{F}}$ – Function of activation.

Decoding function representation maps Y to reconstruct the image X.

$Y\text{ }\!\!'\!\!\text{ }=g\left( Y \right)={{S}_{g}}\left( W\text{ }\!\!'\!\!\text{ }Y+{{b}_{y}} \right)$    (13)

where, ${{S}_{g}}$– Decoding function.

Autoencoder trained and reduced the loss of recreation.

$\theta = {\min\limits_{\theta}{L\left( {X,X^{'}} \right) = {\min\limits_{\theta}{L\left( {X,~gF(x)} \right)}}}}$     (14)

${\hat{P}}_{j} = \frac{1}{m}{\sum\limits_{x = 1}^{m}\left( {a_{j}^{2}x^{i}} \right)}$      (15)

where, $a_{j}^{2}{{x}^{i}}$– Neuron activation j in layer 2 ${{\hat{P}}_{j}}=P$ to impose L1 regularization using sparsity to restructure the image shown in Figure 8.

8.png

Figure 8. Sparse autoencoder model architecture

Algorithm

Step 1: Initialization

{

Step 1.1: Set hyperparameters for WQPSO: Swarm size N, maximum iterations Tmax, inertia weight w, cognitive learning rate c₁, and social learning rate c2.

Step 1.2: Define FMRCNN architecture parameters: Learning rate$\eta$ , batch size B, network weights$\theta$, and activation functions.

}

Step 2: Initialization of Particle Swarm

{

Step 2.1: Initialize particle positions Xi and velocities Vi for i = 1, 2, ..., N within predefined bounds.

}

Step 3. Evaluate Fitness

{

Step 3.1: For each particle i, evaluate fitness using FMRCNN:

Fitness(Xi) = FMRCNN (Xi, θ)    (16)

}

Step 4: Update Particle Positions

{

Step 4.1: Update particle positions using velocity and personal/global best positions:

${{X}_{i}}\left( t+1 \right)={{X}_{i}}\left( t \right)+{{V}_{i}}\left( t+1 \right)$    (17)

Step 4.2: Apply quantum operator ${{X}_{i}}\left( t+1 \right)$:

${{X}_{i}}\left( t+1 \right)=Quantumoperator{{V}_{i}}\left( t+1 \right)$      (18)

}

Step 5: Update Velocities

{

$\begin{gathered}V_i(t+1)=w \cdot V_i(t)+c_1 \cdot r_{1, i}(t+1) \cdot\left(P_{\text {best }, i}(t)-\right. \\ \left.X_i(t)\right)+c_2 \cdot r_{2, i}(t+1) \cdot\left(G_{\text {best }}(t)-X_i(t)\right)\end{gathered}$        (19)

$r_{1, i}(t)$ and $r_{2, i}(t)$ are random numbers between 0 and 1.

}

Step 6: Convergence Check

{

Step 6.1: Repeat steps 3-5 until convergence criteria are met:

Step 6.2: Stop if $t \geq$ Tmax or change in fitness $\leq \epsilon$

}

Step 7: Training FMRCNN

{

Step 7.1: Train FMRCNN using the optimized parameters obtained from WQPSO:

$\theta* = {\underset{\theta}{\text{argmin}}{\sum\limits_{j = 1}^{M}{Loss\left( y_{j},~FMRCNN\left( X_{j},\theta \right) \right)}}}$      （20）

M is the number of training samples, $y_j$  is the ground truth label of sample j, and Loss is the loss function

}

Step 8: Validation and Testing

{

Step 8.1: Validate trained FMRCNN on a validation dataset to assess performance metrics

Step 8.2: Test the model on unseen data for evaluation.

}

Step 9: Fine-Tuning (Optional)

{

Optionally, fine-tune FMRCNN using techniques like transfer or additional training epochs

}

Step 10: Output Results

{

Output the final trained FMRCNN model along with performance metrics

}

This algorithm outlines the integration of Hybrid WQPSO-FMRCNN for enhanced recognition of altered facial expressions in ASD.