Depression Symptom Identification Through Acoustic Speech Analysis: A Transfer Learning Approach

The system architecture proposed for research is as shown in Figure 1, which uses the RAVDESS dataset and GS. We will load the data by reading the audio dataset, then extract the above features from it and determine dependent and independent data. Next, get training and testing sets by splitting the dataset and initializing the parameters for the MLP classifier to train the model. At the end, evaluate the model by calculating the accuracy and analyzing the performance of the classifier.

image003.png

Figure 1. System architecture

The observed accuracy is not satisfactory; hence, optimization is done using hyperparameters for GS. The model is again trained and tested. An improvement in accuracy of up to 71% is observed.

The process for identifying depression symptoms using speech signals is proposed using the following steps:

3.1 Load the speech dataset

Depression symptoms identification using audio datasets RAVDESS and TESS. The datasets are available freely to the public and contain audio files without any background noise. This has the advantage of improving the accuracy of the model.

For the research, the audio datasets used are:

RAVDESS is the Ryerson Audio-Visual Database of Emotional Speech dataset [44]. It contains 7356 files of 247 individuals with different emotions, intensity, and genuineness. The size of the dataset is 24.8GB.

The RAVDESS dataset includes the following data fields:

• audios: A tensor that represents the audio files in wave format.
• modalities: A tensor used to represent different forms of the audio.
• vocal_channels: A tensor used to represent different vocal channels in audio.
• emotions: A tensor that encapsulates a range of emotions associated with the audio file.
• emotional_intensities: Refers to a tensor that represents different levels of emotional intensity.
• statements: These are represented by a tensor that captures the text extracted from the audio.
• repetitions: A tensor that represents either the first or second repetition.
• genders: A tensor used to differentiate between male and female audio.

TESS is the Toronto emotional speech set dataset [12]. It contains 2800 speech files and different emotions such as sadness, anger, fear, disgust, happiness, pleasant surprise, and neutral for each of them.

Two actresses, aged 26 and 64, recited a set of 200 target phrases in the carrier phrase "Say the word _____." Recordings were taken of the set, with each word being portrayed in seven different emotions: anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral. The total number of stimuli is 2800. Two actresses were enlisted from the Toronto region. Both actresses are native English speakers, hold university degrees, and have received musical education. Both actresses' audiometric testing revealed thresholds that fall within the usual range.

3.2 Preprocessing

(1) Feature Extraction (Mel, MFCC, Chroma, ZCR, and RMS)

Librosa is the main library used, along with Pyaudio and Soundfile. In Python, the Librosa library is used to analyze music and audio. It is backward compatible in terms of code readability and modularity. Sklearn is also used to build a model using MLP classifier for emotion recognition using sound files.

The Librosa module is used to convert inputted audio files to digital data. Librosa uses the Mel, MFCC, Chroma, ZCR, and RMS features for this conversion. MFC plays a major role in emotion recognition from speech signals [45]. Mel captures the frequency of the signal represented on the Mel scale, MFCC describes the short-term spectrum of an audio file, the Chroma feature captures harmonic and melodic characteristics based on the pitch of the audio file, ZCR represents the rate of sign changes in the signal for the particular frame, and RMS analyzes the changes in the loudness to extract features from a new audio file.

(2) Relevant feature selection

Emotional features present in the audio dataset are pitch, energy, loudness, and rhythm. Acoustical information is obtained from these during a preprocessing technique called feature extraction. The Librosa module is used to convert data from an audio file into digital form using features such as Mel, MFCC, Chroma, ZCR, and RMS. These speech features contain necessary information that helps with the classification.

3.3 Set the hyperparameters required for GS

Hyperparameter optimization is a technique that involves searching through a range of values to find a subset of results that achieve the best performance on a given dataset. When performing hyperparameter optimization, we first need to define a parameter space or parameter grid, where we include a set of possible hyperparameter values that can be used to build the model. The GS technique is then used to place these hyperparameters in a matrix-like structure, and the model is trained on every combination of hyperparameter values. The model with the best performance is then selected.

The hyperparameters are required to obtain the optimum performance of the architecture. The optimized values of these hyperparameters are mentioned in Table 2 and implemented in experiments as shown in Figure 2.

Table 2. Details of GS hyperparameters

By default, MLP has three hidden layers.

The activation function for the hidden layer used for hidden layers is logistic, which means the logistic sigmoid function returns f(x) = 1/(1+exp(-x)).

For weight optimization, the Adam solver is used. It refers to a Stochastic Gradient-Descent (SGD)-based optimizer. Adam is a perfect solver for datasets with large training samples. It is efficient for both the training and validation phases. SGD updates the weight parameters after each individual training sample. It is faster and can handle large datasets.

Batch_size, when set to "auto," is calculated as min(200,n,samples).

Learning rate determines weight update schedule. The constant value indicates that the learning rate is given by the parameter ‘learning_rate_init’. Its value is 0.001. During weight updation, step size is controlled by the learning rate. This rate is applicable only for Adam Solver with SGD Optimizer.

The maximum number of iterations iterated by the solver is indicated by Max_iterations. It indicates the epoch value for stochastic solvers like SGD or Adam. Epoch means the number of times each data point is used.

3.4 Split the dataset into training and testing sets

The used dataset should be divided into training and testing sets. In machine learning, it’s a crucial step. The model learns from the training set and applies the learned knowledge to the testing set (unknown data). The accuracy at this stage assesses the performance of the model.

3.5 Apply the MLP classifier and train the model

Using MLP, the output is predicted for each sentiment class in the set. The set contains five classes: angry, disgusted, fearful, neutral, and sad. Each of these sentiments is related to depression and can be considered a depression symptom. Hence, if the sentiment of the input audio indicates any of the above sentiments, then the audio indicates the risk of depression.

A neural network is a replica of the human brain with multiple interconnected neurons. The neural network works on a function that uses a scalar product, which is the product of input with weight.

f(x,w)=x₁w₁+….x_nw_n                  (1)

pi_zhu_2024-03-02_163756.gif

Figure 2. GS hyperparameters

where, x_i is input and w_i is weight. The neuron is known as the perceptron, as it performs a human brain-like function of perception. The perceptron is a basic unit of computation that relies on weighted sums, as shown in Eq. (1). Perceptron combines input into a weighted sum, and when the sum crosses the threshold, it fires output.

    $y=\{\begin{matrix}

   \text{Weighted}  \\

   1,\text{ }\!\!~\!\!\text{ if }\!\!~\!\!\text{ }{{\sum }_{\text{i}}}{{\text{w}}_{\text{i}}}{{\text{x}}_{\text{i}}}\text{ }\!\!~\!\!\text{ T}>~0  \\

   0,\text{otherwise}  \\

\end{matrix}$                       (2)

If the weighted sum is greater than the threshold, then it gives the output as 1, otherwise 0 as illustrated in Eq. (2). Such perceptron is applicable for classification tasks. The difference between misclassified data and target data is called loss. To minimize this loss, the optimization function used is stochastic gradient descent. The last function used is the activation function, which can make the decision on the activation of a neuron whether it will fire or not.

A MLP has stacked neurons arranged in layers named input layers, output layers, and one or more hidden layers. In MLP, the neuron can use any activation function arbitrarily. In the model, the logistic sigmoid function is used as an activation function. The sigmoid function returns an s-shaped curve, whereas in the logistic sigmoid function, the return value varies between 0 and 1 instead of -1 and 1. The logistic sigmoid function is given in Eq. (3).

f(x)=1/1+e^-x               (3)

The MLP model is shown in Figure 3.

Figure 3 represents MLP models consisting of an input layer, three hidden layers, and an output layer. The features extracted from the audio dataset act as input. The result obtained from computations of input x_i and weight w_i is propagated to hidden layers in a forward direction; hence, it is also called a feed-forward network (step 1 in Figure 3). Using the activation function, the results of the hidden layer are forwarded as output. Error estimation means the loss calculation (step 2 in Figure 3).

Loss is defined as the difference between targeted output and obtained output. The model is said to be accurately trained or learned when it has a negligible error function. Weight plays a crucial role in learning. Hence, the model has to find the correct weights to minimize the loss. The model has to undergo multiple iterations to get trained. Using the backpropagation algorithm, the model adjusts weights in the network, keeping the goal of the minimum loss function (step 3 in Figure 3).

Backpropagation relies on gradient descent as an optimization function. The weight update process includes propagation in a backward direction from hidden layers to the input layer to minimize the error given by Eq. (4). The weights in the next timestamp depend on the previous timestamp to reduce errors.

$\Delta w\left( t \right)=-\in \frac{dE}{dw\left( t \right)}+\propto \Delta w\left( t-1 \right)$                (4)

The backpropagation algorithm [46, 47] runs through three phases:

Phase 1: Feed-forward phase

This phase propagates from the input layer to the output layer by calculating the weighted sum and applying the activation function.

Phase 2: Back-propagation phase

The exact reverse of phase 1 is backpropagation, which moves from the output layer to the input layer. After calculating the loss function at the output layer, phase 2 updates weights using Eq. (4).

Phase 3: Global Error Phase

This concluding phase finds the effect of weight adjustment using backpropagation on the loss.

3.6 Test the model on a test set

In addition to training, testing is also a crucial step in machine learning. As the dataset is divided into a training set and a testing set, this step evaluates the performance of the model on unseen data in the testing set. The model is learned when it achieves better accuracy during the testing phase. The model is called generalized and is trained for real-world problems.

image007.png

Figure 3. Back propagation in MLP

The performance of both algorithms, GS and TL, is evaluated using a confusion matrix and classification report.

Confusion Matrix: The table that evaluates the performance of the classification task is the Confusion Matrix. The target labels are compared with the predicted labels. It provides a detailed view of the classification and accuracy of the model for all classes. When the class distribution in a given dataset is imbalanced, the confusion matrix is the tool to assess the performance of the model. It is represented in Table 4.

Classification Report: The classification summary after the evaluation of the model is given by the classification report. By using the information in the confusion matrix, it defines evaluation metrics like precision, recall, F1 score, accuracy, and support for all the classes. These parameters are defined using Eqs. (5) and (8). Support specifies the number of instances in the dataset that belong to a specific class. The Classification Report is a useful tool for multi-class classification (more than two classes). It identifies how correctly each class is predicted by the model.

Figure 5 indicates results after executing algorithms 1 (GS) and 2 (TL). The results are displayed using the Confusion Matrix and Classification Reports for both datasets, RAVDESS and TESS. Algorithm 1 (GS) is applied to the RAVDESS dataset, and Algorithm 2 (TL) is applied to the TESS dataset. For GS, precision ranges from 52 to 75%, F1-score ranges from 52 to 78%, and accuracy is 71%. For TL, all metrics—precision, recall, F1-score, and accuracy—range from 99 to 100%.

Table 5 shows the results obtained by each class (anger, sadness, disgust, neutral, and fearful) in all 3 phases of the experiment. The phases are experiments performed on the RAVDESS dataset using an MLP classifier and without GS. The second phase of the experiment is on the same RAVDESS dataset using GS. And in the third phase, experiments are carried out using the same trained model for different datasets (TESS. The evaluation parameters used are precision, recall, F1 score, and accuracy. The corresponding plot is shown in Figure 6.

Table 4. Confusion matrix

qqtu_pian_20240302132707.png

Figure 5. Confusion matrix and classification report for TL from RAVDESS to TESS

image013.png

Figure 6. Results of 3 phases

The values in Table 5 clearly indicate that when no hyperparameter optimization is applied and only an MLP classifier is used on the RAVDESS dataset, the sentiments of five classes are classified with a poor accuracy of 65 to 68%. Even precision, recall, and F1 scores are also poor. A slight improvement is observed when parameter tuning is performed on the same classifier with the same input dataset. Extremely high improvements in accuracy are observed when TL is applied to the trained model and the dataset is changed to TVSS. All the evaluation metrics using both techniques (GS and TL) achieve performance of more than 98%. The findings highlight the potential of TL as a powerful approach for improving the prediction of depression using sentiment analysis on audio data. The improvement in accuracy obtained is highlighted by the green color in the graph shown in Figure 6.

Table 5. Evaluation metrics of 3 phases