Hybrid Deep Learning Framework for Autism Detection in Children Using Facial Emotion Recognition

The medical conditions collectively referred to as autism spectrum disorders (ASD) are varied in several children’s. They are distinguished by a certain level of relationships and communication problems. Atypical patterns of behavior and operations, such as trouble switching between activities, attention to detail, and odd responses to emotions, are additional traits. People with autism have different needs and abilities, and these might change over time [1]. Early diagnosis and intervention are critical for improving developmental outcomes and quality of life for affected children. However, traditional diagnostic methods rely heavily on subjective behavioral assessments by clinicians, which can lead to delays or inconsistencies in diagnosis, particularly in resource-constrained environments [2]. Emotional signs and face emotions are crucial behavioral indicators for diagnosing ASD. Facial study reveals that children having autism frequently display unusual facial expressions, decreased eye interactions, and restricted emotional reactivity [3, 4]. Through the use of computer intelligence and deep learning strategies, these non-verbal indications offer a useful and approachable way to identify individuals for autism [5, 6]. Children with autism may have limited tastes or behaviors, as well as difficulties interacting and interacting with others. They may also employ different attentional, gestural, and psychological strategies. Even while there is now no approved treatment for ASD, early identification is essential for timely medication to lessen symptoms and assist the kid in developing the skills they will need to survive within the future [7]. The researchers' study looked at the potential use of facial features in children to detect autism. Their findings indicate that children with autism display a distinct set of facial characteristics that distinguish them from children without autism. The features include an exceptionally big top face with widely spread eyes and an unusually short central facial area that encompasses the edges of the cheekbones and mouth [8]. contrasting adolescents without autism in the second row with autistic in the first row. The differences in facial features between the two groups are depicted in Figure 1, which was extracted from the Kaggle database. The automation of the detection of ASD using image-based analysis has showed potential through the latest innovations in DL and artificial intelligence (AI). Convolutional neural networks (CNN) among others have proven to be highly effective at extracting intricate visual clues from images of faces [9]. The model's capacity to identify socioemotional deficiencies that are typical of ASD is significantly improved by combining emotion identification with autism screening.

tu_pian_1.png

Figure 1. Visualization of dataset categories: Autistic vs. Non-autistic image samples

In order to identify autism based on facial phrases, this study suggests a hybrid deep learning model which integrates several pretrained CNN models with an ensemble classification approach. Both MobileNetV2 and DeepFace are used to retrieve and evaluate emotion aspects, which allows the framework to read emotional reactions in addition to classifying autism. The goal of the suggested method is to use facial image research to produce an improved and comprehensible testing tool for autism screening. The main contributions can be summarized as:

• To design a robust model for facial emotion recognition in autistic children.
• To extract deep facial features using EfficientNetB0, ResNet50, and MobileNetV3Small.
• To implement an ensemble learning classifier for autism detection.
• To integrate DeepFace for real-time emotion recognition and fusion with autism prediction.

The remainder of this paper is organized as follows: Section 2 presents related work. Section 3 describes the proposed methodology, including feature extraction, ensemble classification, and emotion recognition. Section 4 discusses experimental results and performance evaluation. Section 5 concludes the paper and outlines future research directions.

This section presents the proposed hybrid DL Model for autism detection and emotion recognition and analysis. The methodology comprises five primary components: (1) feature extraction using pre-trained CNN, (2) ensemble classification for autism prediction, (3) emotion-based clustering and classification, (4) real-time emotion detection using DeepFace, and (5) a fusion strategy to integrate both autism and emotion information into a unified prediction system. A high-level system model is outlining the key stages, including multi-CNN feature extraction, ensemble classifier, MobileNetV2 emotion classification, and DeepFace integration. Additionally, the feature- and decision-level fusion strategy for generating the final autism + emotion prediction.

Dataset Description: In this study two dataset namely ASD and ASD with emotion have been used which is freely available [23, 24].

ASD Dataset: The ASD dataset contains the training and testing directories which also contains autistic and non-autistic subdirectories. For training, and testing forms also have two subdirectories: one for people with autism and another for people without.

Figure 1 illustrates representative facial images from the dataset, showcasing both autistic and non-autistic children. These samples reflect variations in facial geometry, expressions, and visual cues, which are critical for training the deep learning models to distinguish between the two categories. The diversity in lighting, pose, and emotion enhances the robustness of feature extraction and generalization during model training.

Figure 2 presents the class-wise distribution of images in the dataset. It highlights the number of samples available for each class autistic and non-autistic that ensure a balanced or imbalanced of the dataset. This distribution is crucial for evaluating model fairness and guiding the use of cross-validation and class balancing techniques during training.

tu_pian_2.png

Figure 2. Distribution of dataset

ASD with Emotion Dataset: This dataset is emotion-specific, pre-processed images of autistic kids which contain six type of facial emotional images such as “Natural”, “anger”, “fear”, “joy”, “sadness”, “surprise”. The dataset consists of total 833 images. Each class of Natural of 48, Anger of 67, Fear of 30, Joy of 350, Sadness of 200 and Surprise of 63.

Data Preprocessing: When collecting data for utilization in proposed ML models and other types of investigation, data processing becomes a crucial step. It aids in ensuring that errors and inconsistencies are eliminated along with ensuring the data is formatted consistently. Whenever the data is used for classification or other kinds of investigation, this may produce superior results. In the instance of the ASD dataset, image processing was used to enhance the evaluation outcomes. The ASD dataset was preprocessed using following distinct methods.

The first method involved resizing each image in the collection. Resizing images in DL models enhances the model's effectiveness in a number of ways. By reducing the strain on the machine during training and inference, it first increases its computational effectiveness by accelerating computation and consuming fewer resources. As a result, the entire model may convergence faster when training. The model gains robust features over a range of dimensions when images are enlarged, which enhances its generalisation and situational recognition capabilities. Inputs must be resized to uniform sizes in order for models to be used in output. It will help with integrating and offer uniformity over a wide range of possibilities. Reduced sizes of images reduce memory requirements and improve model adaptability to real-world conditions through the use of efficient data augmentation approaches. This tactic is particularly important in scenarios with minimal memory because smaller image sizes result in lower storage demands. Image size reduction improves computing efficiency, speeds up training, improves generalizability, and increases deployment flexibility, among other advantages. This was accomplished by comparing the outcomes of proposed ML models using $224~\times ~224$. The images were resized at two distinct configurations using Python programming.

By eliminating the influence of images of various levels, this standardization guarantees consistent and reliable model training. Since normalization avoids issues like disappearing or inflating gradients which arise when images have extremely wide frequency ranges, it accelerates up resolution during training. Furthermore, maintaining numerical consistency enhances the model's potential to pick up important traits. By lessening the impact of high values and illumination fluctuations, the process increases the model's adaptability to different lighting conditions. Normalization also facilitates the optimal use of pre-trained models, which are usually created on datasets having uniform input. Image normalization enhances the effectiveness, strength, and generalization capabilities of proposed ensemble model, allowing them to operate better over a range of scenarios.

The dataset completed these preprocessing techniques to enhance the investigation's outcomes and guarantee that it was in the optimal shape for ML along with additional analysis techniques.

Feature Extraction: In this study, the feature extraction process forms a critical foundation for representing facial images with high-level abstract patterns that are beneficial for distinguishing between autistic and non-autistic children. We utilized a multi-model feature extraction approach, that used the discriminative capabilities of three existing pretrained CNN models: EfficientNetB0, ResNet50, and MobileNetV3Small. These models are pre-trained on the ImageNet dataset and used here as fixed feature extractors by discarding their classification layers and retaining only the convolutional base with global average pooling $\left( include\_top=False,~pooling='avg' \right)$. Each image is first resized to $224\times 224$ and then preprocessed using the respective preprocessing function of each model to match the expected input distribution. For every image, a feature vector is extracted from the output of the final pooling layer of each CNN. These vectors are flattened and concatenated to form a unified representation for further classification. Figure 3 shows architecture, the feature extraction process can be formally represented as follows.

tu_pian_3.jpg

Figure 3. Architecture of autism with emotion detection

Let, $I=\left\{ {{I}_{1}},{{I}_{2}},...,{{I}_{n}} \right\}$ be the set of facial images, $M=\left\{ {{M}_{1}},{{M}_{2}},{{M}_{3}} \right\}$ be the set of pre-trained models, where, ${{M}_{1}}$ is EfficientNetB0, ${{M}_{2}}$ is ResNet50, ${{M}_{3}}$ is MobileNetV3Small.

${{\phi }_{j}}\left( {{I}_{i}} \right)$ be the feature extraction function of model ${{M}_{j}}$ applied on image ${{I}_{i}}$, $Pr{{e}_{j}}\left( {{I}_{i}} \right)$ be the preprocessing function applied to ${{I}_{i}}$ for model ${{M}_{j}}$. Then the final feature vector ${{F}_{i}}$ for image ${{I}_{i}}$ is computed as:

${{F}_{i}}=\left[ {{\phi }_{1}}\left( Pr{{e}_{1}}\left( {{I}_{i}} \right) \right)\parallel {{\phi }_{2}}\left( Pr{{e}_{2}}\left( {{I}_{i}} \right) \right)\parallel {{\phi }_{3}}\left( Pr{{e}_{3}}\left( {{I}_{i}} \right) \right) \right]$          (1)

where $\parallel $ denotes the concatenation operator. All extracted feature vectors ${{F}_{i}}$ are combined into a single feature matrix $X\in {{R}^{n\times d}}$, where $d$ is the total dimensionality of concatenated features from all models. The corresponding labels $y\in {{\left\{ 0,1 \right\}}^{n}}$ are binary, indicating non-autistic (0) and autistic (1) classes. This ensemble feature strategy effectively captures diverse hierarchical representations and enriches the feature space, enhancing the downstream classifier’s ability to discern subtle facial patterns associated with autism.

Ensemble Classification: After obtaining the high-dimensional feature vectors through the feature extraction process, we employ an ensemble learning approach to improve the classification performance for detecting autism based on facial features. Ensemble methods combine the predictive capabilities of multiple base classifiers to produce a more robust and generalized model. In this work, we construct a soft voting ensemble classifier composed of three diverse learning algorithms:

Support Vector Machine (SVC) with a Radial Basis Function (RBF) kernel to capture non-linear decision boundaries,

Gradient Boosting Classifier (GBC), a powerful ensemble tree-based learner that builds models sequentially to correct errors from previous models,

Multi-Layer Perceptron (MLP), a feed-forward neural network that captures non-linear patterns through multiple hidden layers.

Figure 4 shows the proposed ensemble model that each classifier is independently trained on the same training dataset and outputs class probability estimates. The final predicted probability for each class is computed as the average of probabilities predicted by all base classifiers, and the class with the highest average probability is selected as the final output (soft voting).

tu_pian_4.jpg

Figure 4. Proposed ensemble model

To ensure robust evaluation, we adopt Stratified K-Fold Cross-Validation with $k=5$, preserving the class distribution in each fold. This method enables consistent estimation of model performance across different data partitions and reduces variance in performance metrics.

Let, $X\in {{R}^{n\times d}}$ be the feature matrix, $y\in {{\left\{ 0,1 \right\}}^{n}}$ be the label vector, $C=\left\{ {{C}_{1}},{{C}_{2}},{{C}_{3}} \right\}$ be the set of classifiers, where, ${{C}_{1}}$ is the SVC, ${{C}_{2}}$ is Gradient Boosting Classifier, ${{C}_{3}}$ is Multi-Layer Perceptron.

For a given test instance ${{x}_{i}}\in {{R}^{d}}$, let the probability that classifier ${{C}_{j}}$ assigns to class $c\in \left\{ 0,1 \right\}$ be: ${{P}_{j}}\left( c\mid {{x}_{i}} \right)$ is the probability estimate from classifier ${{C}_{j}}$. The final ensemble prediction is given by soft voting:

${{P}_{ensemble}}\left( c\mid {{x}_{i}} \right)=\frac{1}{3}\underset{j=1}{\overset{3}{\mathop \sum }}\,{{P}_{j}}\left( c\mid {{x}_{i}} \right)$           (2)

${{\hat{y}}_{i}}~=arg\underset{c\in \left\{ 0,1 \right\}}{\mathop{\max }}\,{{P}_{ensemble}}\left( c\mid {{x}_{i}} \right)$            (3)

This ensemble formulation allows the model to leverage the strengths of individual classifiers while minimizing their weaknesses, leading to improved generalization and more reliable autism detection from facial expressions.

Emotion-Based Processing

tu_pian_5.png

Figure 5. Cluster of emotion

To enhance the autism detection model with emotional cues, we incorporate facial emotion features into our analysis. These features are extracted from an additional autism emotion dataset, which contains images of children's facial expressions labelled with emotion categories. Initially, we apply data augmentation and normalization using a $ImageDataGenerator$, which rescales pixel values to the range $\left[ 0,1 \right]$ and splits the dataset into training and validation subsets. This step helps ensure that the model generalizes well to unseen data and avoids overfitting. Next, we employ K-Means clustering on the training set to group images based on their underlying emotion features. The K-Means algorithm partitions the dataset into $k$ distinct clusters by minimizing the intra-cluster variance. Figure 5 shows cluster that used k=6 to correspond with typical emotions groups (happy, sad, angry, surprised, fear, and neutral), that serve as the semantic framework for classifying related facial emotions. When emotional data is fed into the DeepFace model for categorization and incorporation into the autism detection mechanism, clustering stage helps to organize the data.

Let, $X=\left\{ {{x}_{1}},{{x}_{2}},\ldots ,{{x}_{n}} \right\}\subset {{R}^{d}}$ denote the set of $n$ preprocessed emotion image vectors, each of dimension $d$. $k$ is set of clusters. ${{\mu }_{j}}~\in {{R}^{d}}$ is the centroid of cluster $j$, where $j=1,2,...,k$. The K-Means algorithm seeks to minimize the within-cluster sum of squares (WCSS) objective:

$args~\underset{\left\{ ~{{\mu }_{j}} \right\}_{j-1}^{k}}{\mathop{\min }}\,\underset{i=1}{\overset{n}{\mathop \sum }}\,{{\left| \left| {{x}_{i}}-{{\mu }_{c\left( i \right)}} \right| \right|}^{2}}$               (4)

$c\left( i \right)\in \left\{ 1,\ldots ,k \right\}$ denotes the cluster assignment for sample ${{x}_{i}}$, $\mu c\left( i \right)$ is the centroid of the cluster assigned to ${{x}_{i}}$, $\parallel \cdot \parallel $ is the Euclidean norm. The next DL module is then informed by the cluster labelling from the trained K-Means model, which enables the system to associate particular emotion expressions with possible characteristics associated with autism. MobileNetV2, a lightweight and effective pretrained CNN model created especially for resource-constrained contexts like mobile is then used to process the clustered data. In the behavioral investigation of ASD, the objective of this stage is to extract precise emotional elements from the children's facial images.

tu_pian_6.png

Figure 6. Performance convergence of MobileNetV2 during training: Accuracy and Loss

Figure 6 shows the performance convergence of MobileNetV2 model. The training loss decreases during the initial epochs and stabilizes at a low value that shows effective feature learning and rapid optimization.

DeepFace-Based Emotion Analysis: In this step, we integrate DeepFace, a state-of-the-art facial analysis framework, to perform emotion recognition as part of our autism detection pipeline. After extracting multi-view features using pretrained CNN architectures of ResNet50, EfficientNetB0, and MobileNetV3Small, the image is passed through a trained autism ensemble classifier to detect the presence of autism. Simultaneously, DeepFace analyzes the same image to determine the dominant facial emotion from predefined classes such as anger, fear, joy, sadness, surprise, and neutral. The purpose of incorporating DeepFace is to complement the autism classification with emotional cues. Emotional recognition can provide deeper behavioral insights, especially in children with ASD, where affective expression may differ significantly from neurotypical peers.

Let, $\in {{R}^{224\times 224\times 3}}$: Input facial image. ${{f}_{r}}\left( x \right),~{{f}_{e}}\left( x \right),~{{f}_{m}}\left( x \right)$: Feature vectors extracted from ResNet50, EfficientNetB0, and MobileNetV3Small respectively.

$F\left( x \right)\in {{R}^{d}}$: Concatenated feature vector, $F\left( x \right)=\left[ {{f}_{r}}\left( x \right),~{{f}_{e}}\left( x \right),~{{f}_{m}}\left( x \right) \right]\in {{R}^{{{d}_{r}}~+{{d}_{e}}+{{d}_{m}}}}$

${{C}_{autism}}$: Trained ensemble classifier (Voting Classifier). ${{C}_{deepface}}$: Pretrained DeepFace emotion classifier. ${{\hat{y}}_{autism}}\in \left\{ 0,1 \right\}:$ Predicted class for autism (0: Non-Autism, 1: Autism)

${{\hat{y}}_{emotion}}\in E$: Predicted emotion class, where, $E=\left\{ Anger,Fear,Joy,Sadness,Surprise,Neutral \right\}$

Then:

${{\hat{y}}_{autism}}={{C}_{autism}}\left( F\left( x \right) \right)$, ${{\hat{y}}_{emotion}}={{C}_{deepface}}\left( x \right)$               (5)

The result is:

$Prediction\left( x \right)=\left( {{{\hat{y}}}_{autism}},{{{\hat{y}}}_{emotion}} \right)$            (6)

The effectiveness of the proposed method is further validated through a comparative analysis with existing autism detection approaches, as summarized in Table 4. Traditional methods like Principal Component Analysis (PCA) and handcrafted texture features combined with classical classifiers such as SVM have shown moderate success but are limited by their reliance on manually engineered features and domain-specific constraints. For instance, Smitha K.G. achieved 82.3% accuracy on the JAFFE dataset using PCA, while Haque M.I.U. reported 77.96% accuracy using texture features on ASD data. Deep learning models like VGG16 and VGG19, as implemented by Reddy P.J., performed significantly lower with 54.15% and 51.44% accuracy respectively, indicating limited transferability to autism-related facial cues. On the other hand, Farooq M.S. demonstrated improved performance with SVM and Logistic Regression on adult datasets, achieving 81% and 78% accuracy, respectively. In contrast, the proposed hybrid DL model used CNN-based feature extraction and an ensemble classifier achieved of 84.00% accuracy, outperforming all compared models. The combination of several CNN frameworks to capture various face expressions and the addition of emotion identification via DeepFace and MobileNetV2, which improves interpretability, are responsible for this better performance. These outcomes attest to the suggested framework's resilience and usefulness in actual autism assessments. The outcomes of the experiment validate the efficacy of the suggested hybrid DL technique for facial expression-based autism detection. The ensemble classifier consistently outperformed individual models such as LR, RF, SVM, and MLP based on evaluation parameters. A more thorough facial expression was achieved by the employment of multiple models for integrated deep feature extraction, and the ensemble learning approach improved generalization and decreased bias. Furthermore, a substantial interpretability layer was added by integrating emotion recognition using MobileNetV2 and DeepFace, which allowed the model to evaluate emotional expressions in addition to detecting autism. This multimodal analysis emphasizes emotional variations commonly seen in children with ASD and facilitates a more comprehensive interpretation of behavioral clues. Even though DeepFace did a good job at recognizing prevailing emotions, more training on datasets unique to ASD might improve its domain-specific capabilities. The proposed approach demonstrates a practical and effective solution for supporting early autism screening using readily available image data, combining the benefits of transfer learning, ensemble modeling, and emotion-aware classification.