Dysarthric Speech Recognition Using Multiple Speech Features and a Lightweight Deep Convolutional Neural Network

Speech is the natural form of human expression for emotions and thoughts. Speech recognition is thus essential to human-computer interaction [1, 2]. The method of identifying human speech using specific computing algorithms is known as automatic speech recognition (ASR). ASR is widely used in automation, affective computing, language identification, speech pathology, speaker recognition, and biometric authorization. Nevertheless, the effectiveness of ASR applications is limited by speech articulation impairment [3-5].

Speech therapy evaluates and treats communication issues and speech abnormalities. Speech-language pathologists (SLPs), sometimes known as speech therapists, carry out this task [6]. Dysarthria is a motor speech condition that develops when the muscles that provide speech signals weaken, are damaged, or are paralysed. Patients with dysarthria have difficulty regulating their voice, tongue, and complex words [7]. Depending on whether a portion of the neurological system is injured, it makes it challenging to produce and pronounce words. Dysarthria is divided into two categories: central dysarthria, which results from damage to the brain, and peripheral dysarthria, which results from damage to the organs required for producing speech. Moreover, dysarthria may be split into acquired and developmental forms. Both before and after childbirth, brain injury may contribute to the development of dysarthria, such as cerebral palsy. A brain injury brings on Dysarthria that develops later in life. Acquired dysarthria may be brought on by a stroke, Parkinson's disease, or a brain tumour. Mumbling, a quiet or soft voice, speaking too quickly or too slowly, and difficulty pronouncing words are all indications of dysarthric disease [8-10].

In recent years, deep learning (DL) methods have been increasingly used in voice-based affective computing and automation systems, as it outperforms more conventional ML-based approaches [11, 12]. Language-related parameters have been shown to be important in developing multilingual ASR. Fathima et al. [13] proposed a time delay neural network (TDNN) to represent acoustic information, which yielded a word error rate (WER) of 16.07% but did not provide local-global connectivity in speech attributes. Yue et al. [14] investigated a multi-spectral acoustic model for dysarthric speech recognition (DSR) based on deep neural network (DNN) and Light Gated Recurrent Unit (LiGRU) technology. Data augmentation minimizes data scarcity by introducing speed perturbations, yielding WERs of 11% and 40.6% for healthy and dysarthric voice, respectively. Furthermore, Yue et al. [15] suggested a hierarchical multiple-stream speech model using LiGRU, fully connected Multi-Layer Perceptrons (MLPs), and convolutional neural networks (CNNs). The electromagnetic articulography (EMA) pre-processed data yielded a WER of 4.6% utilising the suggested model. Butterworth filtering to minimise noise and down-sampling to synchronise Mel Frequency Cepstral Coefficients (MFCCs) attributes are both included in the EMA preprocessing.

The DSR faces significant challenges related to data efficiency. To enhance data, Soleymanpour et al. [16] suggested a text-to-audio synthesizer based on the FastSpeech model. A DNN based on acoustic modelling with a Hidden Markov Model is presented for representing speech data. Conventional data augmentation methods primarily focus on time-domain changes in the signal, while the spectral properties remain constant. Liu et al. [17] suggested tempo, speech, and vocal tract perturbations to create a synthetic dataset. The DNN-based DSR system achieves a WER of 25.21% on UASpeech and 5.4% on the CUHK. To demonstrate the relationship between phonemes and dysarthric speech, Shahamiri [18] employed voicegrams. To reduce data scarcity in DSR, the visual data expansion paradigm is employed. On the UASpeech dataset, the Spatial-CNN (S-CNN) offers a DSR rate of 67%. The suggested S-CNN occasionally results in a vanishing gradient issue and offers subpar outcomes for mild dysarthria. Temporal-domain variation in dysarthric voice and environmental noise significantly affects speech understandability. According to Lin et al. [19], deep learning-based voice conversion (DVC) employing phonetic posteriorgrams (PPGs) yields more stable results than DVC-Mel in noisy environments.

According to Kodrasi and Bourlard [20], harmonics-to-noise ratio (HNR), MFCC, shimmer, fundamental frequency, and jitter performed worse than sparsity in the spectral-temporal domain using the Gini index for the DSR. Spectral sparsity performs more effectively than temporal sparsity. Additionally, Khodrasi's [20] use of CNNs enabled them to learn the temporal spectral properties of voice. The TEFS surpassed the classic Short-time Fourier Transform-based spectrogram of voice signals. The time-frequency CNN (TF-CNN) was studied by Kodrasi [21] to acquire the spectral-temporal characteristics of dysarthric voice. The spectrogram depiction provides distinctive information about changes in pitch, prosody, and intonation compared with normal temporal attributes. Compared to males, the female subjects' DSR performance showed better accuracy. The scarcity and uneven samples in training data leads to class imbalance issue. The TF-CNN provides a better prosodic depiction of dysarthric speech, significantly improving DSR accuracy compared to conventional artificial neural networks (ANNs) [22]. Fritsch and Magimai-Doss [23] suggested a binary classifier using a recurrent neural network (RNN) and a multi-feature classifier using CNN for DSR. It has shown good correlation with text-to-speech (TTS)-generated synthetic speech, but has not yet proven long-term feature correlation. Bhangale et al. [24] suggested that multiple speech features (MSFs) are crucial for depicting speech attributes. It has shown an imperative boost in affective computing applications, but results are challenging for noisy speech. It used only DCNN to characterize multilevel hierarchical attributes but failed to capture long-term temporal dependencies. Thus, from the extensive survey of DSR, the following gaps are identified:

• Poor feature representation of dysarthric voice is caused by more considerable variability in pitch and rapid disparities in the temporal properties of the voice.

• Poor DSR accuracy for the low intelligibility voices due to noise, reverberations, profound changes in voice, pauses, and high volume.

• Poor spectral-temporal depiction of voice provides poor correlation in local-global depiction of the voice.

• Spectral leakage and lower-frequency resolution problems in conventional MFCC lead to inferior features for dysarthric voice.

• Complexity in existing deep learning frameworks, which leads to training time and trainable parameters of systems, and limits the implementation on resource-constrained devices.

Therefore, this article presents a robust DSR system based on DCNN and bidirectional long short-term memory (BiLSTM) to enhance the accuracy of DSR systems for low-intelligibility voices. The key offerings of the paper are summarized as follows:

• Development of MSFs comprising spectral domain features (SDFs), voice quality features (VQFs), and time domain features (TDFs) to boost the distinctiveness of the dysarthric voices.

• Design of DSR system using a hybrid DCNN-BiLSTM, where DCNN supports boosting the hierarchical feature characterization, and BiLSTM offers the superior temporal and long-term depiction of dysarthric features.

The paper is structured as follows: Section 2 elaborates the different speech features utilized for feature extraction. Further, it gives brief details about the DCNN. Section 3 describes on the simulation outcomes for DSR and discusses them. Finally, Section 4 presents a conclusion and possible future improvements.

The acoustic elements of the voice represent its amplitude, frequency, and loudness. A series of discrete SDF, TDF, and VQF is presented to describe the speech dysarthric condition. The wavelet packet transform (WPT), spectrum centroid (SC), shimmer, zero crossing rate (ZCR), spectral rolloff, spectral kurtosis (SK), linear predictive cepstral coefficients (LPCC), pitch frequency, jitter, root mean square (RMS), formants, and their mean and standard deviation (SD) are among the acoustic features that were extracted. The voice stream is processed through a moving average filter before computing different attributes to reduce noise and other interference.

3.1 Mel Frequency Cepstral Coefficient

MFCC offers spectral features analogous to human hearing. The MFCC process is described in Figure 2 [24].

Figure 2. Mel Frequency Cepstral Coefficient (MFCC) feature extraction process

Pre-emphasis standardizes the unprocessed speech stream before MFCC coefficient extraction. The raw dysarthric disordered speech x(n) has noise and disruptions, which are reduced by the pre-emphasis. Additionally, the filtered data is separated into 40-ms frames with a 50% frame shift (20 ms). Overall, 199 frames are produced for 4-second voice when 40-ms frames with 50% overlap are considered. Additionally, the nearest frequency components are gathered together using a Hamming window considering α = 0.46 and NF samples per speech frame of 30 ms, as shown in Eq. (1) [24]. Here, n is chosen such that $0 \leq n \leq N F-1$.

$H(n)=(1-\alpha)-\alpha \cdot cos \left(\frac{2 \pi n}{(N F-1)}\right)$               (1)

The time-domain dysarthric disorder speech signal (s) is then converted into the spectral domain (SP(k)) using the discrete Fourier transform (DFT) using Eq. (2). Eq. (3) gives the DFT power spectrum, which illustrates the peculiarities of the vocal tract. To deliver the voice-hearing perception described in Eq. (4), the voice is processed through 24 Mel-frequency triangular (MFT) filter banks (M = 24). The linear-to-Mel frequency conversion and its inverse are provided by Eqs. (5) and (6). Here, EMT denotes energy of MFT over the frames.

$\begin{gathered}S P(k)=\sum_{n=0}^{N-1} s(n) \cdot H(n) \cdot e^{-\frac{j 2 \pi n k}{N}} \\ 0 \leq n, k \leq N F-1\end{gathered}$                (2)

$S P_k=\frac{1}{N \mathrm{~F}}|S P(k)|^2$       (3)

$E M T_z=\sum_{k=0}^{k=1} \nabla_z(k) \cdot S P_k, z=1,2, \ldots \cdot M$       (4)

$M F T=2595 \, log \left(1+\frac{f}{700}\right)$       (5)

$f=700\left(10^{\frac{M F T}{2595}}-1\right)$       (6)

The cepstral coefficients (L) are estimated by applying the Discrete Cosine Transform (DCT) to the energy signal of the MFT output as given in Eq. (7):

$\begin{gathered}M F C C_i=\sum_{z=1}^M \log _{10}\left(E M T_z\right) \cdot \cos j\left((z+0.5) \frac{\pi}{z}\right) \\ j=1,2, \ldots L\end{gathered}$          (7)

One overall energy feature, 12 MFCC attributes, and 26 1^st- and 2^nd -order MFCC derivatives make up the overall of 39 features offered by the MFCC. The changeover in speech in a dysarthric disorder must be characterized by the derived traits [15, 17].

3.2 Root mean square

RMS $\left(S_{r m s}\right)$ depicts the loudness of the voice, which is typically higher in dysarthric voices than in normal voices, and is calculated using Eq. (8):

$S_{r m s}=\sqrt{\frac{1}{N} \sum_{i=1}^N s_i^2}$           (8)

3.3 Zero crossing rate

ZCR measures the noisiness of the signal by computing the number of zero crossings, which indicate variations in voice due to abrupt changes in pitch and prosody. Eq. (9) is used to estimate ZCR using sgnc function [19]. Within a given time frame (t), the sgnc yields 1 for +ve voice amplitude and 0 for -ve voice amplitude [24].

$Z C R_t=\frac{1}{2}\left(\sum_{n=1}^{N F}(\operatorname{sgnc}(s[n])-\operatorname{sgnc}(s[n-1])\right.$         (9)

3.4 Spectrum centroid

The SIFT SC represents the spectral gravitational centre. It depicts the spectral structure information of the voice. The buildup of broader-frequency elements is indicated by a higher SC value [20]. Eq. (10) offers the magnitude of SIFT $\left(M_t(n b i n)\right)$ across time frame t together with the SC for nbin frequency bins [24].

$S C_t=\frac{\sum_{n=1}^N M_t(n \text { bin }) * \text { nbin }}{\sum_{n=1}^N \text { nbin }}$            (10)

3.5 Spectral rolloff

Spectral rolloff $\left(F_{\text {rolloff }}\right)$ is a degree of spectral structure that offers spectral components beneath which 85% of the magnitude of SIFT is accumulated. Eq. (11) offers an estimate of spectral rolloff [18].

$\sum_{n=1}^{F_{rolloff}} M_t(n $bin$)=0.85 * \sum_{n=1}^N M_t(n$ bin$)$         (11)

3.6 Linear predictive cepstral coefficient

The phonetic depiction of the voice peculiar to dysarthric disorders is represented by the spectral feature known as the LPCC. A total of 13 LPCCs are generated from the linear predictive analysis (LPA) to depict vocal tract attributes [21-24]. The nth samples in an LPA may be predicted using the information from the prior p samples, as shown in Eq. (12):

$\begin{gathered}s(n)=\beta_1 s(n-1)+\beta_2 s(n-2)+\beta_3 s(n-3) +\cdots+\beta_p s(n-p)\end{gathered}$         (12)

where, $\beta_1, \beta_2, \ldots \beta_p$ describe constants over the voice frame that predict the voice element. Eq. (13) is utilized to examine the disparity between the predicted $\hat{s}(n)$ and real sample $s(n)$.

$s(n)=s(n)-\hat{s}(n)=s(n)-\sum_{k=1}^p \beta_k s(n-k)$          (13)

To acquire the exclusive LPCCs, the error $\left(e_n\right)$ between the estimated sample $\hat{s}(\mathrm{n})$ and real sample $s(n)$ is assessed utilizing Eq. (14). Here, m denotes samples in one frame [24].

$e_n=\sum_z\left\lfloor x(z)-\sum_{k=1}^p \beta_k x(z-k)\right\rfloor^2$        (14)

The LPCCs are calculated by estimating solutions for Eqs. (15)-(19).

$\frac{d E_n}{d a_k}=0$, for $k=1,2, \ldots p$         (15)

$C_0=\log _e(p)$         (16)

$L P C C_m=a_m+\sum_{k=1}^{m-1} \frac{k}{m} C_k a_{m-k}$, for $1<m<p$         (17)

$L P C C_m=\sum_{k=m-p}^{m-1} \frac{k}{m} C_k a_{m-k}$, for $m>p$         (18)

3.7 Spectral kurtosis

The series of transients and their frequency-domain positions are provided by the SK, as depicted in Eq. (19). It describes the asymmetry of the voice spectrogram around its centroid, illustrating the influence of energy and pleasantness instabilities in dysarthric disorder on the spectrogram.

$S K=\frac{\sum_{k=b 1}^{b 2}\left(f_k-€_1\right)^4 s_k}{\left(€_2\right)^4 \sum_{k=b 1}^{b 2} s_k}$      (19)

where, $€_1$ and $€_2$ represent the SC and frequency spread, $s_k$ is the frequency elements over k bins, and b1 and b2 are the bins' lower and higher bounds, where the SK of voice is computed.

3.8 Jitter and shimmer

When a vocal fold vibrates irregularly, it causes jitter and shimmer, which are dissimilarities in frequency and amplitude of the dysarthric signal, respectively. The dysarthric disordered sound is characterized by jitter and shimmer, which reflect its roughness, pitch, granularity, breathiness, and hoarseness. The absolute jitter and shimmer values are provided by Eqs. (20) and (21), respectively [24].

$Jitter=\frac{1}{N P-1} \sum_{i=1}^{N-1}\left|T P_i-T P_{i+1}\right|$      (20)

where, $T P_{\mathrm{i}}$ denotes for the time period of successive peak amplitudes in sec and NP depicts the total periods.

$ Shimmer =\frac{\frac{1}{N P-1} \sum_{i=1}^{N P-1}\left|P A_i-P A_{i+1}\right|}{\frac{1}{N P} \sum_{i=1}^{N P} P A_i}$        (21)

where, $P A_i$ signifies the peak amplitude of dysarthric speech.

3.9 Pitch frequency

Pitch is vital in illustrating the vocal element of voice $\left(f_0\right)$. The pitch is computed using variance of the voice peaks obtained from the voice autocorrelation.

3.10 Formants

Formants identify the more energetic peak occurrences in the voice spectrogram. It describes the vocal tract's resonance phenomena, which are predominantly beneficial for characterizing how dysarthric disease affects them. Three formants $f f_1, f f_2$, and $f f_3$ are taken into account for calculation once the formants are computed from the conventional MFCC. To show the formant disparity, the mean and SD of formants is also calculated. Formants $(\mathrm{fm})$, their mean $\left(\mathrm{ff} m_u\right)$, and their standard deviation ($\mathrm{ff} m_\sigma$) are all provided in Eqs. (22)-(24), respectively [24].

$f m=\left\{ff_1, ff_2, ff_3\right\}$        (22)

$\mathrm{ff} m_u=\frac{\mathrm{ff}_1+\mathrm{ff}_2+\mathrm{ff}_3}{3}$        (23)

$\mathrm{ff} m_\sigma=\sqrt{\frac{\sum_{i=1}^3\left(\mathrm{ff} f_i-\mathrm{ff} m_u\right)^2}{3}}$        (24)

3.11 Wavelet packet decomposition features

WPT enables the breakdown of complex information into its simplest components across multiple scales and locations, followed by a highly accurate reconstruction of speech, visuals, music, dysarthria, and patterns. WPT aids in the analysis of speech variability caused by various dysarthric conditions. As indicated in Eqs. (25) and (26), the wavelet packet (WPC) basis function $\Phi_j^i(m)$ utilizing WPT for L-levels is decomposed using the Daubechies (db2) filter.

$\Phi_j^{2 i}(m)=\sum_k \mathrm{ḫ}(k) \Phi_{j-1}^i\left(m-2^{j-1} k\right)$        (25)

$\Phi_j^{2 i+1}(m)=\sum_k \overline{\mathrm{~g}}(k) \Phi_{j-1}^i\left(m-2^{j-1} k\right)$       (26)

where, $\overline{\mathrm{g}}(k)$ and ḫ(k) symbolizes the respective high-pass and low-pass quadrature mirror filters given in Eqs. (27) and (28):

$\mathrm{ḫ}(k)=\left\langle\Phi_j^{2 i}(p), \Phi_{i-1}^i\left(p-2^{j-1} k\right)\right\rangle$      (27)

$\overline{\mathrm{g}}(k)=\left\langle\Phi_j^{2 i+1}(p), \Phi_{j-1}^i\left(p-2^{j-1} k\right)\right\rangle$       (28)

The dysarthric disordered voice is split into subbands at level j using Eq. (29):

$s(m)=\sum_{i k} X_j^i(k) \Phi_j^i\left(m-2^j k\right)$       (29)

where, $X_j^i(k)$ K^th coefficient is at the ith packet at the j level. Eq. (30) denotes the local wavelet energy:

$X_j^i(k)=\left\langle s(m), \Phi_j^i\left(m-2^j k\right)\right\rangle$       (30)

The wavelet packet coefficient (WPC) $X_j^i(k)$ symbolizes the weights of the local WPT denoted by $\Phi_j^i\left(m-2^j k\right)$ as provided in Eq. (31):

$X_j^i(k)=\left\langle s(m), \Phi_j^i\left(m-2^j k\right)\right\rangle$        (31)

Eq. (32) gives a different WPT set for the L level.

$X_L(k)=\left[\begin{array}{c}X_L^0(k) \\ X_L^1(k) \\ \cdot \\ \cdot \\ X_L^{2^{L-1}}(k)\end{array}\right]$         (32)

For each sub-band, seven statistical features are computed, including the mean, variance, median, SD, SK, and energy of each wavelet packet, to capture spectral alterations in the voice. The 3-level decomposition of the voice produces a total of 56 WPT characteristics.

For DSR, the DCNN receives the feature depiction (Feat) as input (a total of 715 features) as in Eq. (33):

$\begin{gathered}\mathrm{MSF}=\left\{M F C C_{1-39}, x_{r m s}, Z C R_{1-199},\right. \\ L P C C_{1-13}, S C_{1-199}, F_{\text {rolloff }}, W P T_{1-56} S K_{1-199}, \\ \left.\text { Shimmer }_1, \text { Jitter }, f_0, f m_{1-3}, \mathrm{ffm}_u, \mathrm{ffm}_\sigma\right\}\end{gathered}$          (33)

3.12 Deep convolution neural network–bidirectional long short-term memory model

Figure 3 provides the process flow of the suggested DSR scheme. The DCNN helps improve spatial correlation, multilevel hierarchical depiction, and correlation between local and global attributes of speech [25-27]. The 1D-DCNN accepts the input of 715 voice features comprising SDF, TDF, and VQF. The conv layer provides the correlation in features (MSF) using learnable filters $(\omega)$. The conv operation is provided by Eq. (34) where z denotes convolution output and σ symbolizes the ReLU. The DCNN consists of three layers with 64 filters in the 1st layer, 128 filters in the 2nd layer, and 256 filters in the 3rd layer. Increasing the number of convolutional layers enhances multilevel correlation and hierarchical feature representation by providing spatial connectivity. The ReLU accelerates training, enhances non-linearity, and reduces the risk of vanishing gradients by substituting negative neurons with zero, as given in Eq. (35):

$z(n)=\sum_{m=0}^{i-1} \operatorname{MSF}(m) \cdot \omega(n-m)$            (34)

$\sigma(z)=\max (0, z)$         (35)

The output of the 3^rd ReLU layer is flattened and provided to the BiLSTM layer with 50 units. The BiLSTM provides contextual information in voice by modeling long-term dependencies in the DCNN representation of the voice features in both forward and reverse directions. The forward LSTM $\left(\overrightarrow{h_t}\right)$ and backward states $\left(\overleftarrow{h_t}\right)$ of BiLSTM are provided in Eqs. (36)-(38):

$\overrightarrow{h_t}=\operatorname{LSTM}_{\text {forward }}(\sigma(z))$        (36)

$\overleftarrow{h_t}=\operatorname{LSTM}_{\text {backward }}(\sigma(z))$        (37)

$h_t^{\text {BiLSTM }}=\left[\overrightarrow{h_t}, \overleftarrow{h_t}\right]$        (38)

The BiLSTM output is provided to the fully connected (FC) layer to offer the linkage between all neurons of the network, as shown in Eq. (39).

$y_{j k}(x)=f\left(\sum_{i=1}^{n_H} W_{j k} x_i+w j_0\right)$        (39)

where, $x_i$ denotes 1 D flattened feature vector, $w_0$ symbolizes a bias, $w$ describes the weight matrix, $f$ depicts a nonlinear activation function (NAF), $y$ stands for the output of NAF, and $n_H$ offers hidden layers. Finally, the SCL provides the likelihood of the output, where the class label is determined by the maximum likelihood score, as shown in Eqs. (40)-(42) [15]:

$ʉ_i=\sum_j h_j w_{j i}$        (40)

$\mathrm{բ}_i=\frac{\exp \left(\mathrm{ʉ}_i\right)}{\sum_{j=1}^n \exp \left(\mathrm{ʉ}_j\right)}$        (41)

$\hat{y}=\arg \max _i \mathrm{բ}_i$        (42)

where, $h_j$ is the weight of the next to last layer and $w_{j i}$ describes the weights of SCL and the next to last layer, ʉ_i is the input of SCL, բ_i is the likelihood of the class label, and $\hat{y}$ is the predicted class.