Robust Emotion Recognition Multimodal Using an Optimized Cross-Modal Data Fusion Framework

Emotion recognition is becoming one of the most important research areas within the fields of affective computing and HCI; this can be seen in the wide array of applications available for intelligent assistants, mental health evaluations, social robotics, and multimedia comprehension. Multimodal emotion recognition (MER) uses a variety of heterogeneous sources (speech, facial expressions, textual clues, etc.) to obtain more accurate indicators of emotional status than unimodal approaches. However, modeling the complex interactions between different modalities, temporal dependencies, and noise inherent to each modality remains one of the significant challenges faced by researchers in this area.

While many recent papers on multimodal learning have concentrated on the alignment of sequences and learning representations of them. Deep Learning models for emotion recognition using multimodalities [1] were discussed in few literatures. Few researchers demonstrated that paradigm of using transformers captures relationships across modalities in both temporally and semantically relevant contexts. From this initial research, several recent reviews have outlined how dominant the use of transformer-based techniques has become within MER, but also highlighted the remaining technical challenges faced by researchers regarding the optimization of fusion strategies and handling computational complexity [2]. In addition to transformer-based techniques, contrastive learning techniques are shown to help improve cross-modal semantics, though these techniques often require large amounts of data and careful selection of sampling strategies [3].

MER has been studied using fusion-centric models. Fusion-based methods, such as the graph convolutional networks architecture [4], leverage graph structures to capture intermodal relationships while also introducing an extra layer of complexity due to their reliance on multiple models. Low-rank fusion techniques such as LMF [5] and MM-LMF [6] reduce the number of parameters and computational load needed for modeling multimodal interactions. However, they still have limitations when trying to accurately model the complex nature of emotions as a result of their reduced representational capabilities.

Dynamic fusion networks [7-8] allow for the adaptive weighting of modalities based on temporal cues; however, they require the specification of fusion heuristics that limit their ability to generalize to other situations. Newer modality-specific representation learning and hierarchical message-passing have also provided an alternative means of retaining emotional semantics better than previously discussed methods. While self-supervised multi-task learning frameworks, such as the study [9], provide enhanced robustness for features, they also add complexity to the training process. On the other hand, hierarchical multimodal message-passing models utilize increased intermodal communications but lack the scalability necessary for practical applications. Hierarchical multimodal message-passing models [10] allow for greater cross-modal communication, but they sacrifice the potential for scalability. Overall, the benefits of adaptive fusion and the need to address the ongoing challenge of developing a single, unified model capable of jointly optimising spatial feature extraction, cross-modal attention and temporal modelling merits continued research in this area.

Overall, these methods demonstrate the need for adaptive fusion and highlight the continued limitations of current technology regarding the joint optimization of spatial feature extraction, cross-modal attention, and temporal modeling.

In response to these issues and challenges, we propose OF-CMAER-CAT based on our findings: the Optimized Fusion–Based Cross-Modal Architecture for MER. OF-CMAER-CAT employs CNNs as spatial encoders, an attention mechanism for cross-modal fusion, and Transformative networks to model both temporal and contextual information. The primary difference between this work and previous works lies in the joint optimization of both the individual representational characteristics for each of the modalities and also the interactions that occur between them via an adaptive-fusion scheme in a manner that allows it to work very well for emotion identification in very diverse conditions as well as in very noisy and heterogeneous settings. OF-CMAER-CAT has been evaluated through extensive experimentation and performance evaluation, and the results have shown that OF-CMAER-CAT outperforms existing fusion-based and transformer-based MER models.

The main contributions of this work are summarized as follows:

• Designing an OF-CMAER-CAT, an optimized fusion–based cross-modal architecture that seamlessly integrates CNN-based spatial feature extraction, attention-driven cross-modal interaction modeling, and Transformer-based temporal context learning for robust MER.
• An adaptive cross-modal attention mechanism is introduced to dynamically weight inter-modal dependencies, effectively suppressing modality-specific noise and redundancy while enhancing emotionally salient features across modalities.
• Unlike existing fusion-centric or transformer-only approaches, the proposed model jointly optimizes modality-specific encoders, fusion layers, and temporal modeling components within a unified end-to-end training paradigm.
• Extensive experiments conducted on benchmark MER datasets demonstrate that OF-CMAER-CAT consistently outperforms state-of-the-art methods in terms of accuracy, precision, recall, and F1-score.

The rest of the paper is structured as follows. Section II provides the existing literature. Section III describes the presented OF-CMAER framework. Section IV reports the experimental details, datasets, preprocessing, and evaluation metrics. Lastly, Section VII summarizes the paper and implications for future research directions.

OF-CMAER-CAT is an optimized fusion-based cross-modal architecture to develop robust and accurate MER using CNN, attention and transformers. The model will be designed to ensure effective fusion of heterogeneous modalities, which include visual facial cues, speech signals, and physiological/textual information, while minimising the effects of modal imbalances, temporal misalignments, noise in the data and cross-modal semantic inconsistencies. In contrast to standard early or late fusion methodologies, OF-CMAER-CAT provides an architecture that combines CNN feature extraction, cross-modal attention alignment, and transformer contextual modeling to provide for adaptive learning of inter-modal dependencies and dynamic optimization of the input contribution of each modal for fusion.

Figure 1. Design diagram of proposed OF-CMAER-CAT framework

The overall design shown in Figure 1 is comprised of a few key elements as discussed below. The proposed Optimized Fusion-Cross-Modal Architecture for Emotion Recognition using CNN, Attention, and Transformer Networks (OF-CMAER-CAT) employs a fully integrated process for learning noise-resilient emotional representations from heterogeneous multimodal data streams. First, independent multimodal raw inputs of visual frames, audio signals, and physiological signals are processed separately through CNN-based modality-specific encoders, which extract both high-level temporal and semantic representations as feature embeddings. The encoders reduce dimensionality and noise while preserving modality properties. After getting the extracted embedding sequences, the Cross-Modal Attention Alignment (CMAA) module aligns the heterogeneous modalities in a common semantic space with scaled dot-product attention, which allows attention given to each modality by the remaining modalities based on emotion relevance.

After alignment, the Adaptive Weight Optimization Module (AWOM) utilizes a set of statistical feature descriptors to produce an estimate of the reliability of each modality. It normalizes the estimated reliability score using softmax normalization to learn the normalized weights for the modalities. This process indicates how much informative modalities contribute to the fused representation compared to the contribution of unreliable or noisy modalities. In addition to this inter-modality consistency enhancement, a contrastive learning objective is utilized that pulls semantically similar modality embeddings together in the latent space while pushing semantically dissimilar modality embeddings apart.

Next, the adaptively weighted and contrastively aligned features are input into a Transformer-based contextual modeling block that combines multi-head self-attention to capture long-range temporal dependencies as well as cross-modal contextual interactions. In doing so, this helps to improve emotion discrimination by modeling small temporal transitions between different emotions. Finally, after the fused representation has been processed through a feed-forward classification head, emotion categories will be predicted based on the output of a softmax layer. To keep the entire network stable and generalizable across multiple datasets, the joint training process uses a multi-task loss function that contains: classification loss, contrastive loss, reconstruction loss, entropy regularization, and weight decay.

To extract meaningful features from different sources of information, each of these sources of input is processed separately by a dedicated Convolutional Neural Network encoder. CNN has been shown to effectively capture not only local spatial characteristics but also temporal variations and modality-specific emotional indicators. A compact latent embedding representation, while at the same time preserving the important emotional attributes contained in the emotion dataset, is achieved through the use of an Encoder that takes raw input from the various types of data collected from a sensor. Redundancy in the data and the amount of processing that goes into fusing all the data from each sensor will be minimised through use of the Global Pooling technique to create two-dimensional (2D) representations of the sensor's data, providing retrospective views of both the time course of the data and its spatial distribution. These 2D representations will serve as an input to the next components of the algorithm for fusing modalities together, as defined by Eq. (1).

$\mathrm{F}_i=E_i\left(M_i ; \theta_i\right), \mathrm{F}_i \in R^{T_i \times d}$                         (1)

Here, $M_i$ is the raw input for modality $\mathrm{i},\, E_i$ is the encoder for modality i with a set of parameters $\theta_i, \,T_i$ is the number of time points for modality i, and d is the number of dimensions in the latent (hidden) feature space. The resulting encoded feature for each modality is represented by the matrix $\mathrm{F}_i$ in $R^{T_i X\,d}$, where the encoding of $\mathrm{F}_i$ is typically a large number of-dimensional time series. A global average embedding of $\mathrm{F}_i$ into a single vector $\overline{\mathrm{f}}_i$ is computed using Eq. (2), which simplifies the calculations performed during the process of fusing each modality, and provides a mechanism to establish optimal alignment between the global vectors obtained from each modality.

$\overline{\mathrm{f}}_i=\frac{1}{T_i} \sum_{t=1}^{T_i} \mathrm{~F}_i\left[t, R^{T_i}\right]$                    (2)

Here, temporal feature vector represented by Fi$[t;]$ at time t, number of time steps, $T_i$, and pooled embedding of modality i or $F i[R(T i)$; ] are used to compute the query, key and value matrices for the cross-modality attention by utilizing linear projections through the attention mechanism. Through this process, the model can find relational contextual connections between modalities and gather supportive features from other modalities using Eq. (3).

$Q_i=\mathrm{F}_i W_Q, K_i=\mathrm{F}_i W_K, V_i=\mathrm{F}_i W_V$                    (3)

Here, $\mathrm{F}_{\mathrm{i}}$ is a matrix containing the encoded features and $\left(\mathrm{W}_{\mathrm{Q}}, \mathrm{W}_{\mathrm{K}}, \mathrm{W}_{\mathrm{v}}\right)$ is an $R^{d X d K}$ type learnable projection at attention dimension dk. The matrices $\mathrm{Q}_{\mathrm{i}},\, \mathrm{K}_{\mathrm{i}}$, and $\mathrm{V}_{\mathrm{i}}$ are of the same dimension as $\mathrm{W}_{\mathrm{Q}}, \mathrm{W}_{\mathrm{K}}$, and $\mathrm{W}_{\mathrm{V}}$, respectively, or $R^{d X d K}$. Scaled Dot-Product Attention computes a score of attention between modality i and modality j to allow modality i to focus its attention only on pertinent features for modality j. Hence, the interaction of modalities and their connections to the emotions will be enhanced through Eq. (4).

$\operatorname{Attn}_{i \leftarrow j}=\operatorname{softmax}\left(\frac{Q_i K_j^{\top}}{\sqrt{d_k}}\right) V_j$                    (4)

Here, Query, Key, Value $\left(Q_i, K_j, V_j\right)$, attention dimension $d_k$, and Softmax(.) are used to normalise similarity scores (cues) in attention-weighted representations $(Attn, (i, j))$.

The attention is directed toward capturing complementary aspects of emotions between audio stimulus (e.g. talking) and visual stimulus (e.g. gaze) while filtering out irrelevant and contrary emotional stimulus (e.g. depressed, happy, angry) that occur within one or both attributes. By employing the CMAA mechanism, the model enhances the degree of accuracy and reliability with which it can produce representations of emotions, thus increasing the level of cross-modal cohesion and stability. The attentional outputs of each modality partner are aggregated into a single, aligned feature vector $\tilde{F_i}$ during the Cross-Modal Aggregation process. The result is a complete integration of all three modalities' complementary information|as represented in Eq. (5).

$\widetilde{\boldsymbol{F}}_i=\sum_{j \neq i} \alpha_{i j} \,A t t n_{i \leftarrow j}$                      (5)

Here, $\boldsymbol{A} t t n i \leftarrow j$ is the attention output from modality $j$ to modality $i, \alpha i j$ is the normalized weight for modality $j$ with respect to modality $i$, and $\tilde{F_i}$ is an $\mathrm{R}^{\wedge}\left(T_i \times d\right)$ aligned feature. The Attention Coefficients calculation provides weights for all partner modalities in order to assign importance to the most similar modality to the target modality as shown in Eq. (6). This is the basis for adaptive fusion in cross-modal interactions.

$\alpha_{\mathrm{ij}}=\frac{\exp \left(\phi\left(\overline{\mathrm{f}}_{\mathrm{i}}, \overline{\mathrm{f}}_{\mathrm{j}}\right)\right)}{\sum_{\mathrm{k} \neq \mathrm{i}} \exp \left(\phi\left(\overline{\mathrm{f}}_{\mathrm{i}}, \overline{\mathrm{f}}_{\mathrm{k}}\right)\right)}, \phi(\mathrm{u}, \mathrm{v})=\mathrm{u}^{\top} \mathrm{W}_\phi \mathrm{v}$                  (6)

Here, $\overline{\mathrm{f}}_{\mathrm{i}}$ and $\overline{\mathrm{f}}_{\mathrm{j}}$ are global embeddings; $W_\phi$ is a learnable weight matrix; $\phi(u, v)$ is the compatibility function returning a scalar similarity score; and $\alpha_{ij}$ is normalized across all partner modalities. The Gated Residual Fusion method allows for cross-modal features $\tilde{F_i}$ to be combined with the original modality $\boldsymbol{F}_i$ via the gating function $\sigma(G i)$. This enables the selective use of cross-modal information while maintaining the inherent salt-and-pepper characteristics of modality-specific features as shown in Eq. (7).

$\mathrm{F}_{\mathrm{i}}^{\mathrm{a}}=\operatorname{LayerNorm}\left(\mathrm{F}_{\mathrm{i}}+\sigma\left(\mathrm{G}_{\mathrm{i}}\right) \odot \widetilde{\mathrm{F}}_{\mathrm{i}}\right)$                     (7)

Here, the sigmoid function is represented by the symbol σ, as well as G. For G_i (MLP gate vector), and the combined features from fusion post-layer normalization are referred to as $F_i^a$. Adaptive Weighting methods use a number of pre-activation score s_i to evaluate each of the modality's importance utilizing Eq. (8) and subsequently yielding adaptive weightings for final weighted fusion, thereby ensuring that quality modalities receive higher contributions to the final representation.

$s_{\mathrm{i}}=\operatorname{MLP}_{\mathrm{w}}\left(\left[\bar{f}_{\mathrm{i}}, \operatorname{var}\left(\mathrm{F}_{\mathrm{i}}\right)\right]\right)$                   (8)

Here, modality feature matrices are denoted as F_i and denote feature vectors characterized by the notation f_i, var(F_i) via vector concatenation. The notation MLPw(.) implies MLP represents a small feedforward network to provide the scalar score s_i for each modality. The contributions of the modalities towards a fused representation are achieved via Eq. (9) with adaptive weightings, allowing the framework to better accentuate the most informative modalities.

$w_i=\frac{\exp \left(s_i\right)}{\sum_k \exp \left(s_k\right)} \in(0,1)$                      (9)

Here, $w_i \in[0,1]$ denote normalized weights $\left(w_i\right)$ and the overall weighting across modalities with respect to the fused representation should equal 1. The binary mask $m i$ is used to account for modalities that may have been corrupted or lost during inference and as a result has been created as part of modality mask missing data. To prevent low-value channels from degrading the quality of the fused output, mode (modality) availability $m_i$ should be defined so that $m_i=1$ means the modality is contributing to the fusion, while $m_i=0$ essentially not contributing.

Once the attention has been aligned, the combined representations of features go through layers of the transformer to learn global temporal correlations and contextual emotional dynamics. The self-attention mechanism of the transformer allows the model to learn relationships across time that are important to accurately classify the more complicated and subtle emotional states. Using transformer-based models also allows the framework to easily handle varying sequence lengths and temporal inconsistencies across differing modalities, which makes the OF-CMAER-CAT framework well-suited for real-world applications of emotion recognition, as shown in Eq. (10).

$\widetilde{\mathrm{w}}_{\mathrm{i}}=\frac{\mathrm{m}_{\mathrm{i}} \mathrm{w}_{\mathrm{i}}}{\sum_{\mathrm{k}} \mathrm{m}_{\mathrm{k}} \mathrm{w}_{\mathrm{k}}+\varepsilon}$                    (10)

Here, $w_i$ are the compatible modality weights that are calculated using softmax with respect to the weight for each modality, $m_i$ is the modality-specific input mask. The weighted modality fusion process produces a single weighted modality-specific fused embedding $h_{\text {fused }}$ of the aligned and weighted modality-specific features of the input by using Eq. (11) to obtain the final weighted fusion of the modalities.

$\mathrm{h}_{\text {fused }}=\sum_{\mathrm{i}} \widetilde{\mathrm{w}}_{\mathrm{i}} \operatorname{Pool}\left(\mathrm{F}_{\mathrm{i}}^{\mathrm{a}}\right)$                   (11)

Here, the fused embedding $h_{\text {fused}}$ in $\mathbb{R} d$ represents the combined alignment of all of the input's features according to the respective weights assigned to each feature during the weighted modality fusion process. As a result of the weighted modality fusion, the fused embedding is sent to a ReLUactivated projection layer that is responsible for transforming it into a compact projection space for classification purposes and adds a non-linear element of classification with respect to the fused embedding as shown in Eq. (12).

$\mathrm{z}=\operatorname{ReLU}\left(\mathrm{W}_{\mathrm{p}} \mathrm{h}_{\text {fused }}+\mathrm{b}_{\mathrm{p}}\right)$                 (12)

Here, the key components include weight matrix (W_p) and bias vector (b_p) in addition to the class softmax layer comparing probabilities between emotional classes from a fused representation as determined by Eq. (13).

$\hat{y}=\operatorname{softmax}\left(W_{\text {cls}} z+b_{\text {cls}}\right)$                 (13)

Here, the number of emotional classes is given by C, while y in RC represents predicted probabilities of C emotional classes based on a softmax layer applied to C classes emotional classes attributes. Cross-entropy loss represents the standard method of assessing how far predicted probability distribution differs from the actual class labels during training based upon the difference in the predicted and true labels calculated via Eq. (14).

$\mathcal{L}_{c l s}=-\frac{1}{N} \sum_{n=1}^N \sum_{c=1}^c y_{n, c} \log \hat{y}_{n, c}$                      (14)

Here, $y_c$ represents true class label (1 for true class; 0 for not) while $\hat{y}_{n, c}$ is the Probability Distribution of predicted Probabilities that Class c belongs to the sample, while C represents the total number of Classes.

Adaptive weight optimisation modules combine contrastive loss with Softmax optimisation process to align cross-codes between definitions and encode data into one binary code so to pull together positively sampled examples of definitions into feature space lattice structure while pushing negative definitions away from these positively sampled examples to enhance cross-code pathway as shown in Eq. (15).

$L_A=\exp \left(\frac{\operatorname{sim}\left(\bar{f}_i^n, \bar{f}_{j(i)}^n\right)}{\tau}\right)$                     (15)

Here, the similarity score between anchor feature and all of its corresponding feature candidates is called $L_A$ and the normalisation factor dividing $L_A$ so as to allow comparison of score across associated feature candidates for a reference feature is called $L_B$ consists of summation of Similarity Scores amongst all candidate features found attached response to anchor feature candidates and where attached refers to cross-path of said similarity scores as shown in Eq. (16).

$L_B=\sum_q \exp \left(\frac{\operatorname{sim}\left(\bar{f}_i^n, \bar{f}_q^n\right)}{\tau}\right)$                      (16)

Here, cosine similarity between the two feature vectors denoted by sim(). It refers to the total number of training samples or batches you are using to compute the loss value denoted with the symbol N, and q is the position that is going to be used when calculating the Normalization for the Batch Samples. In the loss calculation, the exponential exp() will increase the value of the difference in cosine similarities to increase the penalty for bad relationships; the natural log log() will be used to calculate the contrastive loss at the end as shown in Eq. (17).

$\mathcal{L}_{\text {con }}=-\frac{1}{N} \sum_{n=1}^N \sum_i \log \frac{L_A}{L_B}$                    (17)

Here, ($u, v)$ refers to a pair of samples that are positively related, and $f 1^n, f 2^n$ is the reconstructed vector associated with each sample. The term $\tau$ represents the temperature parameter controlling the sharpness of the output vector. The entropy regularization term quantifies the diversity or uncertainty in the normalized weights distribution Eq. (18).

$\mathcal{R}_{\text {ent }}=-\sum_i \widetilde{W}_i \log \left(\widetilde{W}_i+\varepsilon\right)$                 (18)

Here, R_ent is the entropy regularization term where w is the normalized weights and ε is a constant used to prevent the evaluation of ln(0).

AWOM serves as a mechanism to dynamically regulate the impact of each modality. AWOM assigns weights to each learned modality based on feature reliability, variance, and discriminative power. Therefore, modalities that are adversely impacted by noise, occlusion, and/or missing data would have these weights automatically down-weighted, while more informative modalities would be weighted more heavily. The aggregate of the final fused representation is achieved by performing a weighted aggregation of the features, thus resulting in a balanced and optimized multimodal fusion. In addition, an entropy-based regularization technique has been introduced to prevent any single modality from dominating and to promote fair contributions across the different modalities. The penalty applied to the weights through the entropy regularization term as Eq. (19).

$\begin{gathered}\mathcal{L}_{\text {total}}=\lambda_1 \mathcal{L}_{\text {cls }}+\lambda_2 \mathcal{L}_{\text {con}}+\lambda_3 \mathcal{L}_{\text {rec}}+\lambda_4 \mathcal{R}_{\text {ent}}+\lambda_5 \|\Theta\|_2^2\end{gathered}$                    (19)

Here, $\lambda_1, \lambda_2, \lambda_3$ are hyperparameters for each of the loss functions of each loss to indicate how each loss type affects final model training. The results of $L_{c l s}, L_{c o n}$ and $L_{r e c}$ represent the cross-entropy loss or the classification loss, contrastive loss or the A and P loss, and reconstruction or R loss respectively. To perform the tri-modal fusion classification block fusion, the visual, audio, and physiological (V, A, P) features are fused together into one shared embedded space for classification as specified in Eq. (20).

$z_{\text {fusion }}=z_V \, \oplus z_A \, \oplus z_P$                   (20)

Here, ⊕ represents the concatenation of all vectors for V, A, and P features so that z_fusion remains the only feature representative of these three. layer normalization (LN) normalizes the feature representations, resulting in stable and large training datasets. The Feedforward Projection (FP) layer is defined through the feedforward layer with ReLU as the activation function to create a higher level of abstraction from the normalized fused feature set per Eq. (21).

$h_{f f}=\operatorname{ReLU}\left(W_{f f} L N\left(z_{(\text {fusion})}+b_{f f}\right)\right.$                   (21)

Here, the weight and bias matrices involved with $W_{f f}$ and $b_{f f}$ are both learnable, while the $h_{f f}$ variable contains the output from the feedforward layer. The predicted emotion will be derived from which class corresponds to the largest logit as described in Eq. (22).

$\hat{y}=\operatorname{argmax}\left(\mathrm{W}_{\mathrm{o}} \mathrm{h}_{\text {drop }}+\mathrm{b}_{\mathrm{o}}\right)$                  (22)

Here, $W_o$ and $b_o$ represent the matrices of trainable weights and biases, respectively, $h_{\text {drop}}$ represents both the vector of input features and the vector of output logits for this model. It converts the numerical outputs of the model into a discrete decision that will subsequently be used for the evaluation and/or inference of the model.