Dual-Stream Transformer with Reinforcement Learning-Based Attention Optimization for Robust Speech Signal Classification

Today, Speech signals have a major impact on communication. Speech is one of the most natural and effective modes of human communication [1]. Speech classification technique has developed as an important field of human-computer interface. In this classification, the audio signals are anaylzed by providing vital data based on the content of speech signals. It is used for hands-free interaction with the machines with minimal effort [2, 3]. Therefore, speech signal classification becomes a fundamental research area in various applications like emotion recognition, speaker identification, human–computer interaction and speech-to-text (STT) systems. Despite significant progress, accurately modelling the complex and non-stationary nature of speech signals remains a challenging task [4].

Speech signals are heavily time-varying and multi-scale in nature. There are three important steps carried out in speech signal processing. Initially, the processing is carried out to remove noise and to enhance the signal for further handling. Then, the feature extraction is carried out to learn about the signal property. Finally, the classification is performed. Among these stages, feature extraction contributes a critical role. The accuracy of classification mainly depends on the quality of extracted features. To represent speech comprehensively, the classification system should capture the temporal dynamics and spectral characteristics. The nonlinear nature of the signal requires local and global feature extraction for proper classification. The local attributes denote temporal dynamics of the signal. The global attributes reveal the statistical aspects of the signal.

The field of speech signal classification has witnessed substantial progress due to the rapid advancements in Artificial Intelligence (AI) [5, 6]. The AI models like Machine Learning (ML) and Deep Learning (DL), have considerably improved the ability to process and categorise speech signals. These methods have been widely applied in areas like emotion recognition, speaker identification and human-computer interaction. The well-known ML models used for the speech classifications are support vector machine (SVM), Random Forest (RF) and Decision Trees (DT) [7]. Likewise, the DL models like Convolutional Neural Network (CNN), Recurrent Neural Networks (RNN) and Long Short Term Memories (LSTM) are widely applied for speech signal with the consideration of temporal features [8].

Despite these advancements, one of the key challenges in speech signal classification lies in effectively handling the complex and non-stationary nature of speech signals. The existing models struggled with modeling the time-varying characteristics and capturing both spectral and temporal dependencies accurately. To address this research gap, the proposed model introduces a novel hybrid architecture combining advanced feature extraction techniques and a dual-stream transformer network.

In this work, a novel hybrid architecture is suggested for speech categorization. The features like Wavelet Transform, MFCCs, Linear Predictive Coding (LPC), Formant Frequencies, Pitch Contour, and Voice Activity Detection (VAD) are extracted from the input signal to comprehensively learn the speech signal. Then, the dual stream transformer is used to model the complex nature of speech. Also, RL is applied to optimize the attention weights in the network. The proposed model is capable of improving its performance iteratively and well adapts to the intricacies of speech signals. The overall workflow is given in Figure 1.

1.png

Figure 1. Workflow of the proposed classification system

3.1 Feature extraction

Feature extraction involves converting the raw input data into a set of useful features. It captures relevant information and eliminates irrelevant features. The model combines Wavelet Transform, MFCCs, Formant Frequencies, Pitch Contour, and VAD features to represent speech signals comprehensively.

3.2 Wavelet transform features

Wavelet transforms provide a time-frequency decomposition of a signal. In wavelet analysis, the signal is decomposed into different frequency bands using various wavelet functions. Given an input signal x(t), the wavelet transform at scale a and translation b is defined as (Eq. (1)):

$W(a, b)=\int_{-\infty}^{\infty} x(t) \psi\left(\frac{t-b}{a}\right) d t$             (1)

where, $\psi$ is the mother wavelet function, a controls the scale (dilation) of the wavelet, b shifts the wavelet along the time axis. In discrete form, this becomes (Eq. (2)):

$W[a, b]=\sum_{n=0}^{N-1} x_n \psi\left(\frac{n-b}{a}\right)$              (2)

This decomposition provides a set of wavelet coefficients. This coefficient represents the signal's characteristics at different scales and locations in time. After decomposition, the features can be extracted from the resulting wavelet coefficients $\left\{c_1, c_2, \ldots, c_k\right\}$. These features are as follows (Eq. (3) to Eq. (6)) :

Mean:

$\mu=\frac{1}{N} \sum_{i=1}^N c_i$                 (3)

The mean represents the average value of the wavelet coefficients. It gives a measure of the overall signal magnitude at a given decomposition level.

Standard Deviation:

$\sigma=\sqrt{\frac{1}{N} \sum_{i=1}^N\left(c_i-\mu\right)^2}$                 (4)

The standard deviation measures how much the coefficients vary from the mean. It captures the spread or variability of the signal at that level.

Maximum and Minimum:

$\max (c)=\max \left(c_1, c_2, \ldots, c_k\right), \min (c)=\min \left(c_1, c_2, \ldots, c_k\right)$                 (5)

The maximum and minimum values capture the largest and smallest amplitudes in the signal. These help identify peaks or extreme variations in the signal.

Energy:

$ E=\sum_{i=1}^N c_i^2$                (6)

Energy measures the total power of the signal at a particular scale. It is used to quantify the signal’s strength and activity over time.

3.3 Mel-Frequency Cepstral Coefficient

MFCCs are widely used in speech processing to represent the short-term power spectrum of a sound. The process of calculating MFCCs involves several steps. Initially, the Fourier transform is applied to convert the signal into the frequency domain. Then, the Mel-Scale Filter Bank is applied to apply a series of filters which mimic the human ear's perception of pitch. After that, logarithmic compression is used to simulate the loudness perception of the human auditory system. Finally, the Discrete Cosine Transform (DCT) is used to reduce the dimensionality of the features. Given an audio signal x(t), MFCCs are derived as follows:

Mel-Frequency Cepstral Coefficient computation

Short-Time Fourier Transform (STFT): Divide the input into short overlying segments and compute the Fourier Transform for each part as follows (Eq. (7)):

$ X(t, f)=\int_{-\infty}^{\infty} x(\tau) e^{-j 2 \pi f \tau} d \tau$                (7)

where, x(τ) is the input signal as a function of time τ, X(t, f) is the Fourier transform of the short-time segment centered at time t and frequency f, j is the imaginary unit. This computes the frequency content over short overlapping windows of the signal.

Mel-Scale Filtering: Convert the frequency axis to the Mel scale, which is a logarithmic scale more closely related to human hearing. The Mel scale is defined as (Eq. (8)):

$ M(f)=2595 \log _{10}\left(1+\frac{f}{700}\right)$               (8)

where, $f$ is the frequency in $\mathrm{Hz}$, $M(f)$ is the Mel-scaled frequency. This converts linear frequency to a logarithmic scale mimicking human hearing.

DCT for MFCCs: After filtering, the log-magnitude spectrogram is transformed using the DCT (Eq. (9)):

$ \mathrm{MFCC}_n=\sum_{k=0}^{K-1} \log \left|S_k\right| \cdot \cos \left(\frac{n \pi(2 k+1)}{2 K}\right)$                     (9)

where, $S_k$ are the log-magnitude coefficients, $K$ is the number of Mel-filter bins, $n$ is the index of the MFCC. DCT compresses the spectral information into a smaller number of coefficients.

3.4 Linear Predictive Coding features

LPC is used to model the signal as a linear combination of previous samples. It assumes that each sample is a linear function of its past values plus some noise. For a signal x(t), LPC estimates the current sample based on previous samples. The equation is (Eq. (10)):

$ x(t)=\sum_{k=1}^p a_k x(t-k)+e(t)$                  (10)

where, $a_k$ are the LPC coefficients, $p$ is the order of the LPC model, $e(t)$ is the residual error. The formants of the signal which represent resonant frequencies in the human vocal tract. It can be extracted by solving the Toeplitz matrix using the autocorrelation method (Eq. (11)):

$ R=\operatorname{Autocorr}(x(t)) \Rightarrow a=\operatorname{SolveToeplitz}(R)$                 (11)

where, Autocorr$(x(t))$ is the autocorrelation of the signal. It is used to estimate LPC coefficients. SolveToeplitz$(R)$ is the algorithm to solve the Toeplitz system formed by $R$ to get $a_k$ (LPC coefficients). Then, the Formant frequencies can be computed by analyzing the roots of the LPC polynomial.

3.5 Pitch contour features

The pitch of an audio signal represents the perceived fundamental frequency. Pitch tracking involves extracting the frequency of the fundamental tone in a signal. Pitch $f_0$ is determined by analyzing the periodicity of the signal and it can be computed using autocorrelation. Given an audio signal $x(t)$, the pitch $f_0$ is detected by (Eq. (12)):

$ \operatorname{ACF}(\tau)=\sum_{t=0}^{T-\tau} x(t) x(t+\tau)$                  (12)

where, τ is the time shift, and the peak of the autocorrelation function corresponds to the fundamental period.

3.6 Voice Activity Detection

VAD is the process of identifying whether a segment of speech contains actual speech or silence. VAD can be performed by analyzing energy and zero-crossing rate. The energy measures the strength of the signal in a frame (Eq. (13)):

$E=\sum_{t=0}^{T-1} x(t)^2$                (13)

where, $x(t)$ is the input audio signal at time $t$, $T$ is the total number of samples in the frame.

Zero-Crossing Rate (ZCR): Measures the number of times the signal crosses the zero-axis (Eq. (14)):

$Z C R=\frac{1}{T} \sum_{t=1}^{T-1} \mathbb{I}(x(t) \cdot x(t-1)<0)$                  (14)

where, $\mathbb{I}$ is an indicator function, $x(t) \cdot x(t-1)<0$ checks if the signal crosses the zero axis between consecutive samples.

3.7 Spectral features

Spectral features capture the frequency content of an audio signal. It is used to understand the tonal characteristics. The spectral centroid measures the "center of mass" of the spectrum, which gives an idea of the brightness of a sound. It is computed as in Eq. (15):

$\text { Centroid }=\frac{\sum_f f \cdot S(f)}{\sum_f S(f)}$               (15)

where, S(f) is the spectral magnitude at frequency f.

The spectral rolloff represents the frequency below which a specified percentage of the total spectral energy lies (Eq. (16)):

$\operatorname{Rolloff}=\min \left(f: \sum_{f=0}^{f_{\text {max }}} S(f)>\alpha \sum_{f=0}^{f_{\text {max }}} S(f)\right)$                    (16)

where, $f_{\max }$ is the maximum frequency in the frame and $\alpha$ is typically set to $85 \%$ to capture the spectral shape.

3.8 Dual-Stream Transformer

In this work, a Dual-Stream Transformer-based signal processing model is proposed. The main aim of the model is to extract both temporal and spectral features of data. It uses two parallel transformer encoders: one for temporal dependencies and another for spectral dependencies. Then, the outputs are combined through a cross-attention mechanism to refine the learning process. To further optimize the attention mechanism, RL is used to dynamically adjust attention weights based on the model's performance.

The input to the model is a sequence of data, which is first projected into a higher-dimensional space using a linear transformation. This step is used to convert the input data to a more suitable space for the transformer model to process, as follows (Eq. (17)):

$x_{\text {proj }}=W_{\text {proj }} x+b_{\text {proj }}$                 (17)

where, $x$ is the input sequence, $W_{\text {proj}}$ is the projection matrix, $b_{\text {proj}}$ is the bias vector. Then, the projected input is passed to the temporal and spectral streams.

3.9 Temporal Stream Encoder

The Temporal Stream processes the input data considering the sequential, time-based dependencies in the data. The Transformer Encoder is used for this, where each encoder layer is defined as (Eq. (18)):

$\operatorname{Attention}(Q, K, V)=\operatorname{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$                   (18)

where, Q = Query, K = Key, V = Value, d_k  = Dimension of the key vector. The output of each encoder layer is passed to the next layer. The final temporal features are obtained as follows (Eq. (19)).

$h_{\text {temp}}=\text {TransformerEncoder}\left(x_{\text {proj}}\right)$                (19)

Spectral Stream Encoder

Similarly, the Spectral Stream focuses on the frequency-based dependencies. It uses another Transformer Encoder to learn features specific to the frequency domain. The process is similar to the temporal stream (Eq. (20)):

$h_{\text {spec}}=\text {TransformerEncoder}\left(x_{\text {proj}}\right)$                 (20)

Cross-Attention Mechanism

The model combines the outputs using a cross-attention mechanism. This is where the attention weights between the two streams are learned (Eq. (21)):

$\operatorname{Cross-Attention}(T, S)=\operatorname{MultiHeadAttention}(T, S, S)$              (21)

where, T = Temporal features from the temporal stream, S = Spectral features from the spectral stream. The cross-attention mechanism computes attention between these two feature sets. It is mainly used for the model to align the most relevant temporal and spectral components.

3.10 Attention optimization via Reinforcement Learning

In conventional Transformer models, attention mechanisms are used to focus on the most relevant parts of the input sequence during the encoding process. The core idea is that the model assigns weights to different parts of the sequence. This process is mainly used to focus on the most important features. However, the allocation of attention across different parts of the sequence is often fixed during training. In this work, RL is used to dynamically adjust these attention weights based on the model's execution. The goal is to optimize the attention weights to improve the model's learning.

3.11 Reinforcement Learning setup for attention optimization

The RL Agent is responsible for adjusting the attention weights during training. This process involves learning policies that define how attention should be distributed across the sequence. The RL agent performs the following steps:

• State: The state is represented by the hidden state of the model which contains the learned features at any given time.
• Action: The action corresponds to selecting attention heads and deciding which heads should be enhanced. In a multi-head attention mechanism, the RL agent can focus on enhancing the most useful heads.
• Reward: The reward is provided based on how well the model performs. If the model's performance improves after adjusting attention, the RL agent receives a positive reward. If the performance deteriorates, it gets a negative reward.
• Policy Network: The policy network of the RL agent outputs a probability distribution over the available.
• Value Network: The value network estimates the value of the current state.

3.12 Policy network

The policy network learns to output a probability distribution over actions based on the state of the model. The output of the policy network is a set of probabilities $\pi_\theta(a \mid s)$, where $\theta$ are the parameters of the policy network, $a$ is the action (which attention head to enhance), and sis the state (the hidden features of the transformer). The policy network is typically implemented as a fully connected neural network and is trained using policy gradient methods.

3.13 Value network

The value network provides an estimate of the value V(s) of a given state. The value network outputs a scalar that represents how good the current state is in terms of expected future rewards. The advantage is computed as (Eq. (22)):

$A\left(s_t, a_t\right)=R_t-V\left(s_t\right)$                (22)

where, $A\left(s_t, a_t\right)$ is the advantage at time step $t, R_t$ is the return (cumulative future rewards), $V\left(s_t\right)$ is the value of the state at time $t$.

Action: Scaling Attention Weights

The RL agent's action corresponds to scaling specific attention heads. In a typical multi-head attention mechanism, the attention matrix is calculated as (Eq. (23)):

$\operatorname{Attention}(Q, K, V)=\operatorname{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$                  (23)

where, $Q, K$, and $V$ are the Query, Key, and Value matrices, respectively, $d_k$ is the dimension of the key vectors. Each head in multi-head attention learns a different transformation of the input sequence. It is used for the system to concentrate on various parts of the sequence. The RL agent selects a specific attention head $i$ and scales its attention weights by a factor $\lambda$ (Eq. (24)):

$\hat{A}_i=A_i \cdot \lambda$                (24)

where, $A_i$ is the attention matrix for head $i$, $\lambda$ is the scaling factor. The goal is for the RL agent to learn to amplify the most relevant attention heads and reduce the concentration on irrelevant ones.

RL Objective: Policy Loss

The RL agent is trained to maximize cumulative rewards over time. The loss function for training the policy network is based on the REINFORCE algorithm, which aims to maximize the expected return (Eq. (25)):

$L_{\mathrm{policy}}=-\mathbb{E}\left[r_t \cdot \log \left(\pi_\theta(a \mid s)\right)\right]$            (25)

where, $r_t$ is the reward at time $t$ (e.g., classification accuracy), $\pi_\theta(a \mid s)$ is the probability of selecting action $a$ in state saccording to the policy network. This loss encourages the model to select attention heads that lead to high rewards.

Value Loss

The value network is used to estimate how good the current state is. The value loss is computed as the mean squared error between the predicted value $V(s)$ and the actual return $R_t$ (Eq. (26)):

$L_{\text {value }}=\mathbb{E}\left[\left(V\left(s_t\right)-R_t\right)^2\right]$                 (26)

where, $V\left(s_t\right)$ is the predicted value for the state $s_t, R_t$ is the return, i.e., the sum of rewards from time step $t$. This loss is used for the value network to provide better value estimates' Entropy Regularization.

To ensure the RL agent explores different actions and does not get stuck in suboptimal solutions, an entropy regularization term is added. This supports the policy network to maintain diversity in its action selections. The entropy of the policy distribution $H(\pi)$ is computed as (Eq. (27)):

$H(\pi)=-\sum_a \pi_\theta(a \mid s) \log \left(\pi_\theta(a \mid s)\right)$                  (27)

The total RL loss combines the policy loss, value loss, and the entropy term (Eq. (28)):

$L_{\mathrm{RL}}=L_{\mathrm{policy}}+\frac{1}{2} L_{\mathrm{value}}-\beta \cdot H(\pi)$                  (28)

where, $\beta$ is the entropy coefficient that controls the importance of the entropy term. The total loss during training relates to the classification loss and the RL loss. The classification loss is computed as the cross-entropy loss between the predicted and true labels. The RL loss encourages the model to adjust its attention mechanism based on rewards as follows (Eq. (29)).

$L_{\text {total }}=L_{\text {classification }}+\lambda_{\mathrm{RL}} L_{\mathrm{RL}}$               (29)

where, $L_{\text {classification }}$ is the classification loss, $\lambda_{\mathrm{RL}}$ is a scaling factor that controls the importance of the RL loss in the total loss function. The overall Pseudocode is given below:

Pseudocode

Initialize attention weights W_att randomly

for epoch = 1 to MaxEpochs:

    for batch in training_data:

        T, S = TemporalStream(batch), SpectralStream(batch)

        H = CrossAttention(T, S, W_att)

        pred = Classifier(H)

        reward = Accuracy(pred, true_labels)

        W_att = W_att + learning_rate * $\nabla$_W_att(reward)

    if reward converges:

        break

In the proposed Dual-Stream Transformer, standard backpropagation is improved with RL to fine-tune attention weights and improve classification performance. Initially, attention weights $W_{\text {att }}$ are randomly initialized. For each training batch, temporal and spectral features are extracted and combined via cross-attention. The resulting representation is passed to the classifier. The reward is computed based on the classification accuracy or F1-score. Then, RL updates the attention weights in the direction that maximizes this reward. This process repeats over epochs until convergence.

Here, the state represents the combined temporal and spectral features. The action adjusts attention weights based on the reward. Reward defines the batch classification accuracy. It updates based on the equation of $W_{\text {att }}^{(t+1)}=W_{\text {att }}^{(t)}+ \alpha \nabla_{W_{\text {att }}} R_t$ with learning rate $\alpha=0.01$. The training stability is achieved by reward smoothing and gradient clipping with a convergence criterion of $\Delta$ Reward $< \, 0.001$ for 5 consecutive epochs.

3.14 Feature fusion and classification

After weight optimization, the temporal and cross-attended features are fused together. The fusion step concatenates the features from both streams (Eq. (30)):

$h_{\text {fused}}=\operatorname{Concat}\left(h_{\text {temp}}, h_{\text {cross}}\right)$                (30)

The fused features are passed through a fusion layer of a fully connected layer to reduce dimensionality and learn more complex representations as follows (Eq. (31)):

$h_{\text {fused }}=W_{\text {fusion }} \cdot h_{\text {fused }}+b_{\text {fusion }}$               (31)

where, $W_{\text {fusion}}$ is the fusion weight matrix, $b_{\text {fusion}}$ is the bias vector.

Finally, the classification head is applied to the fused features to make predictions(Eq. (32)):

$y=\operatorname{Softmax}\left(W_{\text {class}} \cdot h_{\text {fused}}+b_{\text {class}}\right)$                 (32)

where, $W_{\text {class}}$ is the classification weight matrix, $b_{\text {class}}$ is the classification bias, $y$ is the predicted class.