A Lightweight Hybrid Deep Learning Framework Integrating Multi-Acoustic Features and Convolutional Neural Network and Gated Recurrent Unit Networks for Dysarthric Speech Recognition

DL techniques have recently been extensively adopted in several voice-based automation mechanisms. DL techniques have demonstrated superior performance to conventional ML-based methods. Janbakhshi et al. [11] used a spectro-temporal representation (STR) combined with T-GDA to jointly capture spectral and temporal characteristics. Their approach achieved 96.30% accuracy on the UASpeech dataset. It showed that carefully engineered time-frequency representations can dramatically enhance DSR performance. Bhangale et al. [12] suggested that Mel spectrograms can capture spectro-temporal variations in speech. However, the spectral depiction using Mel filters lacks temporal and long-term correlation in the speech that fails to characterize the voice quality.

Various time frequency depiction schemes have been employed along with DL techniques to boost the DSR accuracy for low voice intelligibility. Fritsch and Magimai-Doss [13] further demonstrated that both recurrent neural network (RNN)-based binary classifiers and CNN-based multi-feature classifiers correlate strongly with synthetic TTS-generated dysarthric speech. According to Chandrashekar et al. [14], time-frequency CNNs yield a more accurate representation of dysarthric speech patterns than traditional neural networks, leading to improved recognition accuracy. Similarly, an article [15] proposed a time-frequency CNN trained using SIFT, SFF, and CQT representations. Although the model performed better on female speakers—likely due to more consistent articulation—training data imbalance was identified as a limitation. Kodrasi et al. [16] used CNNs to capture temporal and spectral structure using TEFS features, outperforming conventional spectrogram techniques. TEFS-CNN achieved accuracy 85.72%, whereas the SIFT-based CNN reached only 69.76%.

Mahum et al. [17] suggested the Swin transformer captures complex multilevel dysarthric features from the voice spectrograms. The multi-view and multi-block modelling help to achieve 98.66% accuracy for DSR.  Its effectiveness is challenging due to complex architecture and lower deployment flexibility on resource limited healthcare devices. Yue et al. [18] suggested multimodal voice modelling that collaborates the acoustic features with articulatory features to enhance the feature depiction. It resulted in WER of 7.6 for TORGO dataset for DSR but the generalization ability of the system is challenging. He et al. [19] suggested cycle GAN model combined with Inception-fusion blocks for generating synthetic sample to reduce the class imbalance issue. It uses error correction scheme to ensure sematic information which resulted in overall WER of 48.69% for low intelligibility voices. Choi et al. [20] proposed that WhisperX-medium provides most satisfactory results for human transcription accuracy for low intelligibility dysarthric voice. The WhisperX models offers better perceptual rating and stronger correlations in ease of understanding and WER. Recording longer voice samples from the dysarthric patient is challenging due to disability and ethical issues which creates data scarcity issue. To tackle this issue, Vijayalakshmi et al. [21] presented data synthesis using text-to-speech models for data augmentation. The FastSpeech2 based TTS has shown WER of 3.49% for Tamil dysarthric voice samples with mild dysarthria. However, the prosodic variation over languages limits the generalization capacity of DSR systems. Vinet et al. [22] suggested Whisper-v3 Model for estimating the intelligibility of dysarthric voices based on phonetic attributes. It offered web based remote applications for DSR what helps to speech language therapists to support the decision making. Chung et al. [23] suggested that Whisper based DSR systems show adaptability to the speaker and needs higher dataset for training. It shows noise robustness, robustness to speech variability, context aware transcription and multilingual support. Its efficiency is challenging due high training and inference time, computational intricacy and need of fine-tuning. To improve ASR effectiveness, Fathima et al. [24] adopted a multilingual TDNN-based model that integrates acoustic modeling with language-dependent cues. With a 16.07% word error rate (WER), their method proved the value of adding language-specific information to feature learning.

Farhadipour et al. [25] employed Gammatonegram spectrograms, a perceptually motivated auditory representation, as input features. Using AlexNet which is well-known CNN architecture. This method achieved 92.3% accuracy, demonstrating the benefits of combining biological features with DL frameworks. Similarly, Kumar et al. [26] utilized raw speech as input to a Residual CNN (ResCNN) framework. This model achieved a modest 75.91% accuracy, indicating better results than simple CNNs with raw audio but still limited compared to feature-rich approaches.

From the survey, various research gaps are identified, such as the high complexity of existing DL algorithms, which limits their feasibility for real-time diagnosis. Existing techniques suffer from poor feature representation, feature redundancy, and inadequate temporal modeling of voice, leading to low DSR accuracy. It has limited accuracy due to low-intelligibility voice and limited modeling of complex dysarthric features. Many methods show limited long-term depiction of the voice signals. These methods cause lower generalization capability.

4.1 System details

Simulation of the proposed method is performed in MATLAB on a personal computer running Windows, equipped with a Core i5 CPU and 4 GB of RAM. Figure 4 shows the proposed network's training accuracy and training loss. DCNN is trained with the Adam optimizer for 100 epochs, a batch size of 64, and an initial learning rate of 0.001. The parameters of the DCNN and different feature extraction algorithms are shown in Table 3.

23-4.png

Figure 4. Training accuracy and training loss graph of proposed deep convolutional neural network-gated recurrent unit (DCNN-GRU)

Table 3. Method parameters and specifications

4.2 Dataset

The results are assessed on the UASpeech dataset. We have considered the digit speech samples of 15 speakers with dysarthria and 13 normal speakers. Age of speakers ranges from 18 to 58 years. Dataset consists of speech samples with four intelligibility levels: high, medium, low, and very low intelligibility. Samples with high intelligibility are considered normal, and those with low, medium, or very low intelligibility are considered dysarthric. We have considered 3,000 normal and 3,000 dysarthric samples. We have considered 70% of the samples for training and 30% for testing [27]. Stability of the experimental results is evaluated using 5-fold and 10-fold cross-validation in both speaker dependent mode (SDM) and speaker independent mode (SIM).

4.3 Performance matrices

The results of the DSR system are evaluated using various qualitative and quantitative measures, including recall, precision, selectivity, negative predictive value (NPV), F1-score, and accuracy, as given in Eqs. (13)–(18), respectively.

$Recall=\frac{\text {True Positive}}{\text {True positive}+ \text {False Negative}}$                       (13) 

$Precision=\frac{\text {True Positive}}{\text {True Positive}+ \text {False Positive}}$                       (14) 

$Selectivity=\frac{\text {True Negative}}{\text {True Negative}+ \text {False Positive}}$                       (15)

$N P V=\frac{T N}{T N+F N}$                       (16)

$F 1-score =\frac{2 \times \text {Recall} \text {× Precision}}{\text { Recall }+ \text {Precision}}$                       (17)

$Accuracy=\frac{T P+T N}{T P+T N+F N+F P}$                       (18)

4.4 Discussion

The confusion matrices for the DCNN, GRU, and DCNN + GRU with multiple acoustic features (MAFs) are shown in Figure 5, using 300 features selected by CBFS for SIM.

23-5.png

Figure 5. Confusion matrix for multiple acoustic feature (MAF)-based dysarthric speech recognition (DSR) for speaker independent mode (SIM)

Table 4 and Figure 6 present performance of the DSR system utilizing different feature sets—SDF, SDF + TDF, SDF + VQF, and MAF (SDF + TDF + VQF)—with three classifiers: DCNN, GRU, and hybrid DCNN-GRU. When only SDF features are used, the DCNN model achieves 86.22% accuracy, 81.00% recall, 90.45% precision, and 85.46% F1-score. GRU slightly improves accuracy to 87.22% and F1-score to 86.71%. The hybrid DCNN-GRU further enhances performance to 88.56% accuracy and 88.08% F1-score. With the addition of temporal information (SDF + TDF), all metrics improve significantly. The DCNN reaches 89.89% accuracy and 90.27% F1-score. GRU attains 91.67% accuracy and 91.83% F1-score. The hybrid DCNN-GRU achieves even higher results with 93.11% accuracy, 93.67% recall, 92.64% precision, and 93.15% F1-score. This confirms that temporal descriptors help to capture speech dynamics more effectively. Further improvement is observed when voice quality features (VQF) are added to Spectral Domain Feature (SDF + VQF). The DCNN records 93.83% accuracy and 93.88% F1-score, GRU increases to 94.50% accuracy and 94.60% F1-score, while the hybrid DCNN-GRU delivers 96.28% accuracy, 96.33% recall, 96.23% precision, and an F1-score 96.28%. These results indicate that VQ-based representations are highly discriminative for dysarthric speech. The best overall performance is achieved using Multiple Acoustic Feature (MAF) fusion (SDF + TDF + VQF). With a DCNN, the system achieves 95.78% accuracy and 95.73% F1-score, while a GRU achieves 97.44% accuracy and 97.47% F1-score. The proposed DCNN-GRU hybrid model achieves the highest scores across all metrics: 98.11% accuracy, 98.33% recall, 97.90% precision, 97.89% selectivity, 98.33% NPV, and 98.12% F1-score. The multi-acoustic feature combined with a hybrid DL architecture provides superior recognition capability for dysarthric speech.

Table 4. Performance of DSR system for different features and classifiers (SIM)

23-6.png

Figure 6. Visualizations of the dysarthric speech recognition (DSR) results for various features and classification methods

Table 5 and Figure 7 compares three optimization algorithms—Stochastic Gadient Descent Momentum (SGDM), Root Mean Square Propagation (RMSProp), and Adaptive Moment Estimation (Adam)—for training the proposed DSR model using standard performance metrics. The SGDM optimizer achieves 97.22% accuracy, 97.00% recall, 97.43% precision, 97.44% selectivity, 97.01% NPV, and 97.22% F1-score. This indicates stable, balanced performance.  The RMSProp slightly improves overall accuracy to 97.50% and F1-score to 97.48%, while showing very high precision (98.20%) and selectivity (98.22%), which means it is more effective in reducing false positives. However, its recall (96.78%) and NPV (96.82%) are marginally lower than SGDM. This indicates a small trade-off in detecting all true dysarthric samples. The Adam optimizer achieves the highest, 98.11% accuracy, 98.89% recall, 98.87% NPV, and 98.13% F1-score. Although its precision (97.37%) and selectivity (97.33%) are slightly lower than RMSProp, Adam maintains a superior balance between sensitivity and specificity. These results demonstrate that Adam offers quicker convergence and improved generalization, making it the highly suitable optimization method for proposed Dysarthric Speech Recognition framework.

Table 5. Comparison of DCNN-GRU-based DSR for different optimization techniques

23-7.png

Figure 7. Visualizations of results of deep convolutional neural network-gated recurrent unit (DCNN-GRU)-based dysarthric speech recognition (DSR) for different optimization techniques

Table 6 presents the accuracy for the different features selected using CBFS. Effectiveness of CBFS-based feature selection is analyzed in the DCNN, GRU, and DCNN-GRU models for DSR. Increasing the number of features (up to 300) improves feature distinctiveness and accuracy. The DCNN-GRU achieves mean overall accuracies of 98.1% with 300 features, 93.1% with 100 features, 87.25% with 50 features, and 96.25% with 200 features. However, increasing the number of features beyond 300 increases redundancy and slightly decreases feature discriminability. The DCNN-GRU achieves accuracies of 97.9%, 97.55%, 96.1%, and 95.25% for 400, 500, 600, and 700 features, respectively. The DCNN-GRU achieves improvements of 0.82% and 2.73% over the GRU and DCNN respectively. The above improvements are for 300 features selected using CBFS. CBFS offers features with lower intra-class correlation and higher inter-class correlation to enhance feature depiction and minimizes redundancy in features.

Table 6. DSR accuracy comparison for different features chosen using CBFS

Table 7 presents the DSR results from 10-fold cross-validation. This shows that all models perform strongly in both SDM and SIM, with consistent improvements as the architectures become more advanced. In SDM, the DCNN model achieves 93.25% mean accuracy ,2.1 standard deviation and a 95% of CI (confidence interval) of [91.45, 95.05], indicating relatively higher variability. The GRU model improves performance to 95.65%, with a reduced standard deviation of 1.6 and a CI of [94.25, 97.05], indicating greater stability. The hybrid DCNN-GRU model achieves the best results with a mean accuracy of 97.15%, the lowest standard deviation of 1.1, and a tighter confidence interval of [96.25, 98.05]. This reflects superior accuracy and robustness. Robust feature selection using CBFS improves model stability, and DCNN-GRU enhances feature distinctiveness, improving the accuracy of the DSR system.

Table 7. Statistical results analysis for SDM and SIM for 10-fold cross-validation

4.5 Ablation study

Table 8 provides ablation study of the system. Principal Component Analysis (PCA) obtained an average accuracy of 95.65% for feature selection. Independent Component Analysis (ICA) improved the performance to 96.20% and Particle Swarm Optimization (PSO) to 96.45%. The proposed CBFS method gave the best result of 98.11% accuracy, which is 2.46% better than PCA and about 1.7-1.9% better than ICA and PSO. When it comes to optimization algorithms, SGDM achieved 97.22% accuracy, RMSProp 97.50%, and Adam the highest, 98.11%. This indicates that Adam has the best convergence capability.

Table 8. Ablation study for DCNN-GRU-based DSR for UASpeech dataset

The learning-rate effect was also significant. The accuracy was 97.20% with a learning rate of 0.0001, and it was increased to 98.11% with a learning rate of 0.001. However, the performance drops to 96.40% at 0.01, and further drops to 93.10% at 0.1. It shows that excessively high learning rates will negatively affect model stability and generalization. The accuracy of recognition was also influenced by the batch size, 96.50%, 96.85%, and 97.50% were achieved by 16, 32 and 128, respectively. The best performance of 98.11% is achieved with a batch size of 64, indicating that a moderate batch size provides a better trade-off between learning efficiency and model generalization.

4.6 Results comparison with traditional Dysarthric Speech Recognition techniques

The DCNN-GRU-based DSR achieves higher accuracy than earlier DL-based DSR methods, as illustrated in Table 9. Effectiveness of proposed model is compared for various popular ML models with and without feature selection.

Table 9. DSR accuracy comparison of DCNN-GRU-based DSR with existing methods for UASpeech dataset

The 715 MAFs provide 82.35% accuracy for support vector machine with radial basis function (SVM-RBF), 92.20% for logistic regression (LR), 92.45% for multi-layer perceptron (MLP), 93.105% for DCNN, 93.505% for GRU, and 95.25% for DCNN-GRU. Feature selection using CBFS yields significant boosts in overall accuracy: 98.11% for DCNN- GRU-CBFS, 97.315% for GRU-CBFS, and 95.45% for DCNN-CBFS with 300 features. The proposed method, which integrates MAFs with a DCNN-GRU, achieves a remarkable 98.83% accuracy on the UASpeech dataset. Compared with Shahamiri [18], who achieved 67.00% accuracy with raw speech and a Spatial CNN, the DCNN-GRU shows a 47.47% improvement. According to Kumar et al. [26], who achieved 75.91% accuracy with a Residual CNN on raw speech, the proposed method shows a 30.22% improvement. In contrast to Farhadipour et al. [25], who achieved 92.30% accuracy with AlexNet on gammatonegram spectrograms, the Proposed Method demonstrates a 7.07% improvement. Even compared to the high-performing method of Janbakhshi et al. [11], which used spectro-temporal features and T-GDA to achieve 96.30%, the Proposed Method still yields a 2.63% improvement. These results highlight the superiority of using a diverse set of features and a deeper DCNN model, making the Proposed Method the most effective approach for DSR in this comparison.