Artificial Intelligence-Based Facial Emotion Valence Detection Model Using Advanced Feature Engineering Techniques

The recognition of facial expressions by FER serves as a vital feature in multiple application domains which include intelligent tutoring systems and HCI and virtual reality and healthcare and affective computing [13]. FER accomplishes emotion detection and classification by analyzing facial characteristics present in image or video data [14]. Research in emotion-aware systems has grown rapidly due to increasing demand because of work done in image processing and facial feature recognition and dimension reduction and feature selection and classification methods.

2.1 Pre-processing techniques in FER

The fundamental stage of FER processing involves removing background noise as well as equalizing lighting levels and image upgrading. Multiple approaches exist to deal with lighting improvements while improving edge detection methods which leads to better recognition abilities [15]. A significant reflection-based method extracts the reflectance picture directly from individual brightness files independently of three-dimensional representation ensuring high performance with diverse illumination settings.

Researchers have adopted image normalization techniques that integrate three components, that include histogram equalization and discrete cosine transformation and enhanced correlation coefficient [16]. The new techniques enhance recognition stability when processing authentic datasets. The combination of cropping and resizing processes with brightness changes helps increase recognition performance although it reduces computational processing needs. When we reduce image sizes to the 34×28 pixel range, researchers have proven that vital facial characteristics remain visible while the data becomes simpler to handle [17].

Edge detection tools represent one of the primary operations in pre-processing applications. Facial contour detection depends on multiple standard operators consisting of Sobel and Prewitt alongside Roberts and Laplacian of Gaussian operators [18]. Feature points become more precise in edge detection using Adaptive Canny algorithms when combined with Active Appearance Models which work effectively under noisy conditions. Decisions involving pattern recognition from direction and filtering lead to stronger feature extraction while removing background interferences [19].

2.2 Feature extraction and dimensionality reduction

The vital second process of FER extracts fundamental visual indicators by identifying various features like eye movements alongside lip curvatures and eyebrow positions. Multiple algorithms use Local Binary Patterns (LBP) to generate textural encodings that maintain their results despite gray-scale transformation monotonicity [20]. Modern research in the field has introduced the variants Compound LBP (CLBP), Local Directional Patterns (LDP), and LDP with variance (LDPv), which achieve effective extraction of spatial along with textural expression information.

The Scale-Invariant Feature Transform (SIFT) descriptors showcase exceptional performance when dealing with transformations in scale along with rotations. Researchers have integrated CNNs with these features successfully to achieve better results with scarce training samples [21]. Research finds that Haar-like features along with Gabor filters assist in drawing spatial frequency and orientation information from the face to improve performance under changing lighting conditions.

GSNMF alongside SLFDA represents sparse methodology to decrease dimensions without compromising discriminatory features. The projection methods deliver effective space reduction which preserves class-specific characteristics so recognition accuracy improves significantly in high dimensional problems [22].

2.3 Feature selection methods

The use of feature selection in FER systems is necessary since it decreases computational load and enhances classifier results through the removal of useless or unnecessary attributes. Various evolutionary and bio-inspired algorithms like Genetic Algorithms (GA), together with Particle Swarm Optimization (PSO), Bat Algorithm (BA) and Whale Optimization Algorithm (WOA) have become widely used in research practice [23].

The algorithms work in association with component simplification approaches, including Discrete Cosine Transform (DCT) and Principal Component Analysis (PCA) to generate combined solutions [24]. Combined implementations of GA-PCA with DCT-PCA achieve optimal results for maintaining both feature compactness and recognition fidelity. BBAE and cat swarm optimization function as improved Binary bat algorithms through which both global convergence and classification accuracy can be achieved.

The current strategies incorporate correlation-based and stochastic optimization models to assess both the correlation between features and their discrimination power in different classes [25]. The combination of these approaches produces more accurate FER system generalizations when dealing with high-dimensional along with noisy data.

2.4 Classification techniques in FER

The classification step of FER pipelines assigns features extracted during the previous operations to defined emotion categories. Most classification processes in FER systems utilize SVM along with k-NN and ensemble models [26-28]. High-dimensional spaces benefit from SVMs because they optimize class margins effectively. Through the integration of geometric or appearance-based features with SVMs the system achieves excellent results for identifying faint facial expressions.

The latest methods in this field utilize DL models through CNNs combined with LSTM networks and RNNs. These types of models demonstrate capability to learn both temporal relationships and sophisticated hierarchical patterns contained in facial information [29-31]. Autoencoder stacks and LSTM-RNN network designs deliver better continuous emotion detection along with greater resistance to noise because of their improved performance measures.

Stability in various datasets becomes more achievable through the combination of CNN learning approaches with logistic regression or deep autoencoding. GANs help create simulated prototypical expressions to add to data availability and minimize intra-class variation issues [32]. The system's performance is improved by manifold regularization methods and dictionary learning techniques because they both minimize intra-class diversity and boost class differentiation levels.

The classification of facial regions represents another approach which divides images into smaller square or triangular sections to capture unique domain features [29]. A focused learning approach occurs due to localized representation methods which minimize data dimensions while producing better performance across different facial types and population groups.

2.5 Research gap

Despite significant advancements, FER still faces several challenges. The current techniques experience difficulties while handling mixed conditions between partial obstructions and differences in lighting and between distinct subjects. The combination of inefficient feature selection with sparse differential representation leads to incorrect classifications when dealing with data of poor or restricted quality. Real-time systems suffer from the drawback of having slow computational procedures.

Peterbal approaches today do not implement strong deep learning models or multi-channel inputs that include speech or physiological signals, thus minimizing their flexibility scope. The accuracy rate of emotion identification is hindered due to ineffective edge detection and optimization methods. One promising approach to addressing these restrictions is to combine recent advances in deep neural networks with optimization methods utilized for feature selection.

The literature review tracks the FER system evolution from different computational stages. Image pre-processing functions provide consistent quality, followed by expression feature capture, then data reduction occurs through dimensionality methods before classification outcomes emotional interpretations. The accuracy of various models appears promising but fixing problems with data variability, along with noise and complexity issues, stands as the most important task. Even though the problem of CNN-based FER has been extensively studied, the current methodologies have several limitations. Numerous handcrafted deep hybrid models are designed based on simple concatenation, and do not face the issue of representational discrepancy between modalities. The current attention models usually do not use FER-specific calibration of micro-expression regions. Moreover, most of the approaches to optimizing rely on a plain SGD or Adam without hyperparameter exploration on the global level. Lastly, many past studies exhibit good intra-dataset accuracy with poor cross-dataset generalisation. The method proposed overcomes such limitations by bringing in; FER-specific attention-residual modeling, theoretically-grounded fusion, and combining optimization and cross-dataset validation.

The AffectNet database serves as the research foundation because it stands as a major dataset suitable for emotion recognition studies. AffectNet was developed by Ali Mollahosseini et al. and comprises over 1 million facial images obtained through internet searches with multi-lingual emotion-related keywords. AffectNet serves as an optimal database for FER model assessment and training through its extensive collection of images featuring numerous lighting conditions and background types and multiple head orientations across various ethnicities across both uncropped and obscured faces.

Every image in AffectNet contains two types of labels: categorical emotional categories along with dimensional values. The categorical annotations consist of eight main emotion classifications which include happy, sad, angry, fearful, surprised, disgusted, contempt and neutral. When analysing face images, manual annotations accounted for 450,000 images to measure their valence range from -1 to +1 while quantifying arousal levels from -1 to +1. Researchers can perform holistic psychological investigations of emotional expression due to two distinct labelling systems built into this dataset.

The AffectNet platform provides boxes that contain facial coordinates together with landmark points, which allows automated face detection and coordinate alignment in preprocessing operations. Multiple high-quality annotations along with the extensive dataset size render AffectNet a more effective tool than both FER-2013 and CK+. This research applies normalization and histogram equalization and custom features extraction techniques to pre-process the inputted dataset prior to D-CNN application. The model achieves enhanced generalization capability through its exposure to the diverse real-world characteristics that exist in the AffectNet database. AffectNet enables the study to achieve robust performance along with practical application for emotion-aware systems running in real-life conditions, such as healthcare security functions and educational environments.

The section evaluates experimentally the proposed model for FER which relies on the AffectNet dataset and D-CNN. The methodology for model training and testing appears first before an evaluation of performance based on standard metrics. The evaluation comprises a comparison with current models which demonstrates the proposed approach's performance capabilities. A cross-validation procedure under controlled conditions was used to conduct the experiments for reliable and consistent performance assessment. The obtained results will serve to identify what the model does well but also what it does not in various practical operational conditions.

In order to provide a just and impartial test, authors add five highly employed baseline models depicting standard machine learning algorithm Support Vector Machine (SVM), Multiple Kernel Learning (MKL), traditional deep learning algorithm Convolutional neural network (CNN), and hybrid characterization feature-deep architectures such as Deep Fusion CNN (DF-CNN), and fully connected neural explainers as Feedforward Neural Network (FNN). They are baselines of both handcrafted-feature and previous CNN variants in the FER literature. In addition, the updated edition includes more state-of-the-art deep FER models (e.g., ResNet-50, VGG-FER, ACN) that would be more rigorous to compare. It has also included cross-dataset testing on CK + and FER-2013 to illustrate the performance in generalization beyond AffectNet.

These comparison paradigms include classical, kernel-based, and deep-models, which allows to conduct the balanced evaluation of the comparative performance of our proposed framework and emphasize the gains, which are made in the context of various algorithmic paradigms.

6.1 Performance metrix

FER models need to be evaluated using a variety of metrics to study the accuracy, strength, and discrimination at class-level. In a bid to offer an overall evaluation, the proposed model is evaluated based on accuracy, precision, recall, F1-score, specificity, and MCC, and also AUC-ROC, and other multi-class metrics, and which would provide a fair evaluation of the baseline methods.

Accuracy: The overall performance accuracy of a model relies upon accuracy measurement. The measure calculates true predictions against the total number of all instances in the model.

$\text{Accuracy}=\frac{T P+T N}{T P+T N+F P+F N}$            (20)

Precision: It assesses how many correctly predicted positive expressions exist among all positive output results. The measure indicates how well the model performs at identifying particular emotional states.

$\text{Precision} =\frac{T P}{T P+F P}$             (21)

Recall: It determines whether the model finds all existing positive instances accurately.

$\text{Recall} =\frac{T P}{T P+F N}$              (22)

F1-Score: The F1-score finds its value through precision and recall harmonically averaged together to create a weighted metric which benefits situations with uneven class distribution.

$\text{F1-Score} =2 \times \frac{\text { Precision × Recall }}{\text { Precision }+ \text { Recall }}$          (23)

Specificity: The model proves its ability to accurately detect non-positive elements through the specificity metric.

$\text{Specificity} =\frac{T N}{T N+F P}$              (24)

Matthews Correlation Coefficient (MCC): A balanced metric named MCC provides performance assessment for all four values within the confusion matrix, which proves helpful for handling imbalanced data sets.

$\mathrm{MCC}=\frac{(T P \times T N)-(F P \times F N)}{\sqrt{(T P+F P)(T P+F N)(T N+F P)(T N+F N)}}$           (25)

ROC Curve and AUC: The Receiver Operating Characteristic (ROC) curve plots the True Positive Rate (Recall) against the False Positive Rate (FPR), where:

$\text{FPR}=\frac{F P}{F P+T N}$             (26)

AUC represents the mathematical value of model probability to categorize positive instances above negative instances randomly selected from a sample.

Here TP = True Positives, TN = True Negatives, FP = False Positives, FN = False Negatives.

Cohen Kappa (κ): Agreement between forecasted and actual classes, not due to chance.

$\kappa=\frac{p_o-p_e}{1-p_e}$          (27)

where, $p_o$ is an observed agreement and $p_e$ is expected agreement by random chance.

Balanced Accuracy (BA): Tunes sensitivity and specificity in the face of class inequity.

$\mathrm{BA}=\frac{1}{2}(\mathrm{TPR}+\mathrm{TNR})$            (28)

Geometric Mean (G-Mean): This is to make sure that the false positives and false negatives are minimized.

$\mathrm{G}=\sqrt{\mathrm{TPR} \times \mathrm{TNR}}$           (29)

FPR and FNR Analysis: The trends in misclassifications between the emotions are further explained by the False Positive Rate (FP / (FP + TN)) and False Negative Rate (FN / (FN + TP)) values.

Table 2, along with Figure 5, displays how various FER methods perform in terms of accuracy measurement through five repeated trials. Traditional methods like SVM and FNN show relatively lower accuracy, with averages around 78–80%. The accuracy performance of MKL and CNN and DF-CNN improves until it reaches 84.55% in DF-CNN. The proposed D-CNN model proved to be superior to all present techniques by delivering the highest accuracy score, which ranged between 93.79% and 94.61% across each test run. The proposed method demonstrates strong practical application because it uses deep convolutional architectures to successfully learn features and make classifications. This enriched feature representation, supported by effective normalization and a robust D-CNN framework, enhances discriminative power and stability across iterations, leading to consistently higher accuracy. The model's persistent leadership proves its capability as an ideal solution for instant emotion detection needing precise accuracy.

Table 2. Comparison of accuracy of existing approach with suggested approach

5.png

Figure 5. Representation of compared accuracy

Table 3. Comparison of precision of existing approach with suggested approach

Table 3 with Figure 6 displays how different FER models performed regarding precision throughout their five test runs. Precision levels of FNN stay consistently low among traditional models while SVM and MKL, together with CNN demonstrate a stable yet moderate performance. DF-CNN achieves higher precision levels which range between 77.64 and 79.21 percent showing enhanced discrimination ability toward features. The proposed D-CNN model produces superior performance to all other models by achieving precision rates from 88.55% to 90.21% which proves its high efficiency in identifying genuine positive expressions. The proposed model achieves improved precision due to its enriched feature representation generated through visual bagging, spatial mapping, and fusion of handcrafted and deep features. These complementary features reduce ambiguity between similar expressions, enabling the D-CNN to make more accurate positive predictions and consistently minimize false positives across iterations.

6.png

Figure 6. Representation of compared precision

Table 4. Comparison of recall of existing approach with suggested approach

7.png

Figure 7. Representation of compared recall

The recall data for different FER models appears in Table 4 and Figure 7 during five evaluation runs. Both FNN and SVM exhibit insufficient recall capability since they fail to detect multiple authentic positive expressions. The recall values from DF-CNN models demonstrate consistent improvement that reaches 82.14% and MKL and CNN both achieve better performances. The improved recall is achieved because the proposed model captures a richer and more discriminative feature space through visual bagging, spatial mapping, and fusion learning. These mechanisms help the D-CNN detect subtle emotional patterns, reducing missed true positives. As a result, the system consistently identifies a larger proportion of actual emotion classes across iterations. Throughout all iterations the proposed D-CNN model demonstrates the best recall results starting from 92.41% which consistently maintained until it reached 94.21%. The model demonstrates consistent performance by lowering false negative rates, thus providing essential capabilities to applications that need high sensitivity in emotion detection.

Table 5. Comparison of F1-score of existing approach with suggested approach

8.png

Figure 8. Representation of compared F1-score

Table 5 alongside Figure 8 demonstrates the F1-score evaluation metrics, which combine precision and recall performance metrics for different FER models during five repeated iterations. The proposed model achieves higher F1-scores because its fusion of handcrafted and deep features creates a more balanced representation of emotional cues. This reduces both false positives and false negatives, improving precision and recall simultaneously. Visual bagging and spatial mapping further enhance discriminability, ensuring consistently strong performance across all iterations. The F1-scores obtained by FNN and SVM traditional classifiers demonstrate weakened performance because they achieve inadequate detection precision and reduced false prediction control. MKL, CNN and DF-CNN display medium to high performance results while DF-CNN reaches up to 81.01%. The proposed D-CNN model provides the highest F1-scores across evaluation runs from 90.42% to 92.10% which demonstrates its capacity for precise and balanced detection. The proposed method demonstrates exceptional ability in preserving classification accuracy throughout different test iterations.

The analysis of Table 6 and Figure 9 shows the specificity results for different FER models across five testing trials to identify proper negative facial expressions. The proposed model attains higher specificity because its fused feature representation effectively separates true negative samples from ambiguous emotional patterns. Visual bagging and spatial mapping enhance structural distinctions, reducing false alarms. As a result, the D-CNN accurately identifies non-target classes, delivering more reliable discrimination and consistent performance across iterations. Hence, FNN and SVM exhibit inferior specificity values because they identify neutral and non-target expressions as positive outcomes more frequently. MKL and CNN together with DF-CNN achieve fair progress in the study but DF-CNN maintains the highest specific rate at 84.10%. The proposed D-CNN model achieves superior performance compared to every other model since its specificity measures from 92.63% to 94.66%. The model displays excellent precision by detecting non-relevant expressions with accuracy, ensuring reliable facial expression analysis.

Table 6. Comparison of specificity of existing approach with suggested approach

9.png

Figure 9. Representation of compared specificity

The performance evaluation of different FER models is presented in Table 7, together with Figure 10, through their respective MCC values across five iterations. MCC serves as an evaluation criterion to assess binary and multiclass classification performance when dealing with class imbalance parameters. FNN along with SVM demonstrates weak performance in terms of MCC scores indicating their inability to establish accurate correlations between predictions and actual results. DF-CNN stands out among the models by reaching an MCC value of 0.73, while MKL and CNN show moderate results. The D-CNN model achieves superior performance when compared to others based on MCC scores that demonstrate predictive strengths from 0.819 to 0.851. The model displays superior performance because it demonstrates effective adaptation to multiple expression classification situations.

Table 7. Comparison of MCC of existing approach with suggested approach

10.png

Figure 10. Representation of compared MCC

Table 8. Comparison of AUC-ROC of existing approach with suggested approach

The AUC-ROC (Area Under the Receiver Operating Characteristic Curve) scores across five iterations provide information about model ability to separate emotion classes according to Table 8 and Figure 11. The classification abilities of FNN and SVM remain at a moderate level, as demonstrated by their AUC value range from 82 to 85%, respectively. The three models MKL and CNN together with DF-CNN generate enhanced performance outcomes whereby DF-CNN delivers the highest result at 90.83%. These results indicate improved classification of emotional categories. The proposed D-CNN model maintains the best AUC-ROC scores which span from 97.86% to 99.35% indicating its superior discriminative abilities. The model effectively recognizes true positives and true negatives because of its superior performance levels.

11.png

Figure 11. Representation of compared AUC-ROC

12.png

Figure 12. Representation of compared Cohen’s Kappa

Table 9. Comparison of Cohen’s Kappa, Balanced Accuracy and Geometric Mean of existing approach with suggested approach

Figure 12 and Table 9 juxtapose Kappa (k) as suggested by Cohen in different emotion recognition methods. The highest κ over 0.89 of the proposed D-CNN models has been found and indicates a better agreement and reliability than the conventional approaches, such as SVM, MKL, CNN, DF-CNN, and FNN.

The comparison of Balanced Accuracy (BA) and Geometric Mean (G-Mean) is provided in Table 9 and Figure 13 based on different face emotion recognition models. The proposed D-CNN is characterized by the highest BA of 94.8% and G-Mean of 94.2, which proves to be a perfect balance of sensitivity and specificity. Conversely, the conventional methods like SVM, MKL, CNN, DF-CNN and FNN are scored relatively low with DF-CNN scoring better among the existing methods (85.5% BA, 84.3% G-Mean). This implies that the suggested D-CNN is more stable and robust in terms of its classification over class imbalance and varied facial expressions.

13.png

Figure 13. Representation of compared Balanced Accuracy and Geometric Mean

6.2 Computational complexity analysis

In order to evaluate the practical viability of the proposed FER model, we evaluate both the complexity of the algorithms and empirical performance. The computational expense of the convolutional backbone is majorly taken over by convolutional layers which require:

$O\left(K^2 \cdot C_{\text {in }} \cdot C_{\text {out }} \cdot H \cdot W\right)$           (30)

where, K is the size of the kernel, $C_{\text {in}}, $C_{\text {out}}$ are input/output channels and H, W are spatial dimensions. Another O(C) global average pooling and channel recalibration overhead is also introduced by the AERB module, and is insignificant compared to convolutional cost.

SIFT + BoVW feature extraction is a handcrafted method which works with the complexity:

$O\left(N_{k p}+D\right)$            (31)

where, $N_{k p}$ is the key point count distracted and D is the length of the descriptors.

Normalization (L2 concatenation) Feature fusion Feature fusion:

$O\left(d_h+d_d\right)$           (32)

where, $d_h$ and $d_d$ refer to handcrafted and deep feature dimensions.

The optimization time of the PSO-Adam hybrid is determined by the fact that PSO conducts global search on $\mathrm{O}(\mathrm{P} \cdot \mathrm{I})$  and Adam conducts an update of O(T) times/cycle.

In general, the suggested structure has moderate computing cost and supports high levels of discriminative capabilities that can be applied in the real-world implementation of the FER.

To measure the actual efficacy of the suggested FER system, sometime measurements are tested:

Training Time (TT) gives the average time taken to pass the training data showing the efficiency of optimization.

Inference Time (IT) time it takes to process a single input image, which represents a real-time application.

Throughput (TH) quantifies processing capacity in a system, which is significant with FER applications of high volume or high streams.

The combination of these metrics can be verified to make the proposed FER framework not only accurate but computationally efficient and scalable.

Table 10. Comparison of Training Time, Inference Time, and Throughput of existing approach with suggested approach

14.png

Figure 14. Representation of compared Training Time and Throughput

15.png

Figure 15. Representation of compared Inference Time

The efficiency of training and throughput (samples processed per second) of various approaches to FER is compared in the Table 10 and Figure 14. Conventional approaches like SVM and FNN take a longer duration to train and to perform worse throughput because they are less parallel and poor in feature-learning. Deep learning based CNNs are more computationally efficient and DF-CNN performance is superior to other existing baselines. The D-CNN proposed is the best-performing one due to its low Training Time (22.1 s) and the highest throughput (233 samples/s), due to its optimized architecture and hybrid learning approach.

The comparison of the inference-time in Figure 15 and Table 10 demonstrates that the results provided by the proposed D-CNN is the fastest (4.3 ms), and all existing models are more expensive. Its optimized architecture can detect emotions much faster in addition to being highly accurate.

The presented D-CNN has significant performance outperformance to the current methods in both computational and recognition measures. Our model has the shortest possible Training Time (22.1s), which means that it converges faster than SVM, MKL, CNN, DF-CNN, and FNN. The best time of inference is recorded in the proposed architecture as well (4.3 ms), which makes it possible to deploy it to real-time situations where quick emotion identification is critical. The throughput is notably larger (233 samples/s), which is over two times more than the optimum baseline, which proves the effective processing large-scale or streaming FER applications. In addition to computational improvements, the proposed model is always more accurate, more precise, has higher recall, F1-score, specificity, MCC, as well as AUC-ROC, which proves strong and credible emotion classification. Generalization over diverse datasets is also enhanced by the addition of AERB optimization, optimal fusion and hybrid PSO optimization-Adam refinement. The combined benefits in the recognition quality and the computation efficiency obviously demonstrate that the presented D-CNN is a better and more viable FER solution than the current ones.