ECG - Based Emotion Detection via Parallel - Extraction of Temporal and Spatial Features Using Convolutional Neural Network

2.1 Dimensional model (valence - arousal model)

It is difficult to model or judge human emotions because individuals express their emotional states differently based on such factors as their subjective feeling and cognitive process. Over the past decades, several scholars have devoted to developing different emotion models for modelling human affective states. In the discrete models, users must select a specified list of word labels to tag/label emotional states in discrete categories for denoting their current emotion. Thereby, the stimuli may evoke mixed emotions that cannot be sufficiently expressed in words since the meanings of the selected words are culturally dependent. Thus, the discrete models need more than one word to specify blended emotions. While in the emotional dimensional models, individuals need to scale affective states in multiple dimensions for classifying emotions. Lately, two popular scales used for classifying emotions are valence and arousal planes. And most of the former studies have focused on using the 2D model to model emotions. So, rather than modeling emotional states of ECG signals as discrete emotions, they can be categorized by level of valence and arousal each (see Figure 2). For the aforementioned reasons, we applied the 2-dimensional Russell's model [20] to facilitate ECG-based emotion recognition. This model demonstrates that affective states are distributed in a 2D space with dimensions of valence and arousal. Valence plane indicates the horizontal axis and reflects the degree of pleasure that extends from highly negative to highly positive, while arousal plane represents the vertical axis and reflects the strength of emotional activity that ranges from low/passive to high/active.

2.png

Figure 2. Valence-arousal model/theory of emotions. X-axis denotes the valence plane, while Y-axis indicates the arousal plane

We can observe from Figure 2 how emotional states are categorized by this model. As we can also note that emotional states mapped in the lower-left quadrant are classified to be “Low Valence-Low Arousal”, whereas in the upper-right quadrant are categorized to be “High Valence-High Arousal”. Regarding the emotional states mapped in the lower-right quadrant are classified to be “High Valence-Low Arousal”, while in the upper-left quadrant are categorized as “Low Valence-High Arousal”. However, by determining the value of valence with arousal for some emotions, we can recognize affective states as four categories. In this study, the recognition task of emotional states is split into a binary classification method. This indicates that, the private and the DREAMER datasets are categorized in a two-category format by assigning a threshold at the average output value that divides the outputs into low and high. Thus, in our dataset, we split the rating scale of 1 to 9 into a binary classification method (low-high) with a threshold value of 5. While for the DREAMER dataset, we divided the rating scale of 1 to 5 into a binary classification method (low-high) with a threshold value of 3.

2.2 Related work

Therefore, several studies have been suggested to provide an ECG based detection system for classifying emotional states.

Among those studies, model selection is a challenging task facing most researchers in the field of affective computing (AC). Besides, feature extraction is also considered a challenging issue and represents a critical stage in the emotion classification.

In addition to the model/classifier selection, we have categorized the feature extraction frameworks for emotion detection into two main groups namely: (1) hand-crafted features, (2) automatic features.

The hand-crafted features need human expertise to extract useful information from ECG signals manually, which include different features such as (time-domain features, frequency domain features, time-frequency features, statistical features, and others). While regarding the automatic-extracted features, there is no need to extract features manually, where deep learning techniques such as convolutional neural networks (CNNs) can automatically extract robust features from raw ECG data. We reviewed different studies based on traditional and deep learning techniques that have been conducted in recent years, as presented below.

Subramanian et al. [21] proposed a Naive Bayes (NB) classifier for classifying emotional states. The authors used ASCERTAIN dataset that composed of 58 participants, and they used 36 emotive movie clips for eliciting affective states based on ECG signals. They also employed various features from ECG data such as frequency features and statistical features. The NB classifier achieved a classification accuracy of 60% and 59% for valence and arousal respectively. Wiem and Lachiri [22] suggested a support vector machine (SVM) classifier for detecting emotional states. They utilized MAHNOB database, which consisted of 24 subjects who participated in this experiment. The affective states were triggered using 20 video clips. Statistical features were extracted from ECG signals for recognizing affective states. The SVM classifier achieved a classification accuracy of 68.75% for valence emotions and 64.23% for arousal emotions. Hsu et al. [23] proposed a Least Squares SVM classifier to recognize and classify emotions based on ECG signals. The affective states were elicited using music from 61 subjects. Time-domain, frequency-domain, and nonlinear features were extracted from ECG signals. Thereafter, the LS-SVM classifier achieved a classification accuracy of 82.78% and 72.91% for both valence and arousal respectively. A study by Katsigiannis and Ramzan [24] introduced a SVM+RBF Kernel classifier for classifying emotional states. The classifier was applied to the DREAMER dataset that consisted of 23 subjects. Besides, the affective states were provoked using 18 various Audio-Video video clips. Heart Rate Variability (HRV) features and PQRST features were used in their proposed study. Finally, the SVM+RBF Kernel classifier attained a classification accuracy of 62.37% and 62.37% for valence and arousal respectively. Baghizadeh et al. [25] proposed a SVM-Polynomial and SVM-Linear for detecting human emotional states. Moreover, the researchers used time-domain, frequency-domain, time-frequency domain, and nonlinear features, afterward the SVM-Polynomial classifier attained a classification accuracy result of 78.07% for valence emotions, while the SVM-Linear achieved a classification accuracy of 82.17% for arousal emotions on the MAHNOB dataset.

In contrast to these five conventional methods for feature extraction, there is a deep learning technique. Deep learning (DL) is becoming an important method in bio-medical signal processing applications.

Santamaria-Granados et al. [26] proposed a developed deep convolutional neural network (DCNN) model for recognition of emotion. They obtained 75% and 76% results of classification accuracy for valence and arousal on the AMIGOS dataset that collected from 40 subjects, while they were watching 16 different video clips. Sarkar and Etemad [27] suggested and developed a self-supervised method based on a convolutional neural network (CNN). They achieved an average accuracy of 85% and 85.9% for both low/high valence and low/high arousal respectively on the DREAMER dataset. Finally, Harper and Southern [28] presented a developed CNN-LSTM model for classifying affective states based on ECG signals. On the DREAMER dataset, the CNN-LSTM model achieved a classification accuracy of 86% for low/high valence emotions.

In this section, before the raw data are fed into the proposed system for detecting ECG-based emotion, they are required to be processed first according to the following steps:

4.1 Filtering

The second step filtered the ECG signal from noise and artifacts for both datasets. The ECG signal is typically contaminated by several types of noise, such as powerline interference and artifacts caused by body movements, respiration, and others which can affect extracting the information related to emotion from ECG signals. Therefore, it has become necessary to eliminate potential powerline interference and baseline drifts before being fed into the model for classifying affective states. As reported in the study [23], we filtered the ECG signal by applying two median filters, the first with a sliding window of length (600 ms) and the second with a sliding window of length (200 ms). Consequently, the resulting signal from the two median filters was passed to a (12-order) low pass filter with a cut-off frequency of (35 Hz) for removing the powerline interference resulted of the electrical network that ranges between (50 Hz - 60 Hz), as mentioned by Cuomo et al., and Berkaya et al. [30, 31].

4.2 Downsampling and segmentation

Concerning our dataset, the third step down-sampled the ECG signal to 250 Hz from 1000 Hz by a factor of 4 and then segmented it into 20-second windows with no-overlapping between the windows as depicted in Figure 6. Accordingly, the segmentation of the ECG signal resulted in 1260 chunks (21 subjects × 60 image) with a length of 5000 (20-sec × 250 Hz) data samples in each chunk ( $x \in \mathbb{R}^{5000}$ ) from 21 subjects.

Regarding the segmentation of the ECG signals of the DREAMER dataset, some of the studies use only the emotional signals without considering the impact of the neutral baseline signals (without emotional activity). ECG signals produced by the same participant under the same stimulations are often different due to the instability of human ECG signals and their sensitivity to some alternations in the surrounding environment. Therefore, in this study, we considered the baseline signals that were recorded through eighteen neutral video clips. As the neutral clips have no emotional activity was shown before each stimuli video clip (containing emotional activity) in order to help the participant, return to the neutral affective state. Whereas, these signals have proved their role and competence in improving the accuracy of emotion classification. Thus, these signals can be used as neutral signals to remove effects from the ECG signals while preserving the emotional content of the ECG data. To do this, we first must split the baseline signals and the experimental/emotional signals into chunks as follows:

1- In the first step we used N = 57-second of the baseline signals after omitting a 4-second out of 61-seconds of it.

2- In the second step we divided the 57-seconds of the neutral signals into 57 1-second chunks, then calculate the average of all chunks, as formulated in Eq. (1).

$\overline{B L}=\frac{\left(\sum_{n=1}^{N} B L_{n}\right)}{N}$     (1)

where, $\overline{B L}$ represents the mean of all the 57 chunks of 1-second each.

3- In the third step we used N=60-second of the emotional/experimental signals, which means that used the last 60-seconds of those signals.

4- In the fourth step we also divided the last 60-second of the emotional signals into 60 1-second segments $\left(E m o_{-} S e g_{i}\right)$.

5- In the fifth step we subtracted the mean value $\overline{B L}$ (computed in step 2) from each 1-second segment of the emotional signals mentioned in step 4, to get baseline removed segments and stack them in a matrix $\left(B L r_{S e g_{i}}\right)$, as indicated in Eq. (2).

$B L r_{-} S e g_{i}=\left(E m o_{-} S e g_{i}\right)-\overline{B L}$     (2)

6.png

Figure 6. The data segmentation, normalization and reshaping process

The aforementioned steps were intended to eliminate the neutral influences from ECG data while maintaining its emotional content. Eventually, the process of segmentation of the ECG signals resulted in 828 chunks (2-channels, 18-videos, 23-participants) with a length of 15360 samples each $\left(x \in \mathbb{R}^{15360}\right)$ (60-second each chunk × 256 Hz).

4.3 ECG segments normalisation

The next step after segmentation is the normalisation process. As aforementioned, there are a lot of factors that can impact the ECG signals such as the DC offset as well as the amplitude variance. To address such problems and to improve the quality of the ECG signals before being fed into the model, we have normalised and converted all the ECG data points to a common interval between [0,1] (Figure 6) through using the (min-max) normalisation method. The (min-max) method can be expressed as follows:

$X_{i}^{n}=\frac{X_{i}^{n}-X_{\min }}{X_{\max }-X_{\min }}$     (3)

where, $\left(X_{i}^{n}\right)$ represents the $\left(i^{t h}\right)$ data point of the $\left(n^{t h}\right)$ sample, while $\left(X_{\min }\right)$ and $\left(X_{\max }\right)$ denotes the minimum and maximum value of the $\left(n^{t h}\right)$ sample respectively.

4.4 ECG segments reshaping

Generally, 2D image signals contain more detailed and high-dimensional information, making them better for model generalisation, while real images have relatively redundant data. As a result, 1-dimensional ECG segments can be reshaped into a 2-dimensional gray-level map as an input to the model to decrease the adverse impacts. Thus, it improves the signal quality from noise comparatively as well as the utilisation efficiency of computing resources. Therefore, in this study, we have used both one and two-dimensional ECG signals as the input of our concurrent multi-channel model for their respective functions of representing various dimensions of information to improve the reliability of ECG data. Figure 6 depicts the process of normalising and reshaping ECG data that is segmented into 20-second windows. Regarding the normalised 1-dimensional ECG segments are kept as inputs to the 1D-CNN model for extracting temporal features. Meanwhile, according to the normalisation rule that the scaled samples values between 0 and 1 corresponding to the gray-level between (light = 0 and dark = 1), the 1-dimensional ECG segments are reshaped to create gray-level maps each. To do that, we take each 50 sample points of the normalised segment and stack them as the first column of the map, and so on to the remaining sample points. Thus, we will get a gray-level map of 100 columns each containing 50 grayscale color values corresponding to the sample points of the signal which were normalised between 0 and 1. Lastly, the generated gray level map can be fed as input to the 2D-CNN model for spatial features extraction.

4.5 Overview of the convolutional neural network (CNN)

This section provides a brief review of the most prominent deep learning methods, where the CNN algorithm is one of the common types of deep learning techniques based on artificial neural network structures, which can be 1-dimensional, 2-dimensional, or 3-dimensional. Generally, the traditional machine learning methods are composed of three different layers as follows: an input layer, one hidden layer, and an output layer, unlike the artificial neural network that has more than one hidden layer in its structure. Therefore, the artificial neural networks (ANNs) structure is inspired by the network of biological neurons of the brain working system that involves several hidden layers [32]. Each hidden layer includes many neurons that act as processing units for the inputs from other neurons in the previous layer, meaning that they provide a new representation of features that were extracted from the input data. The output of each neuron is calculated by the weighted sum of its inputs $\left(X_{i}^{l-1} \times W_{i j}^{l}\right)$), then adding a static value called bias propagated thru a nonlinear activation function as defined in Eq. (4) for the neuron activation and Eq. (5) for its output respectively.

$a_{j}^{l}\left(\boldsymbol{X}^{l-1}\right)=\sum_{i=1}^{N_{l-1}}\left(X_{i}^{l-1} \times W_{i j}^{l}\right)+b_{j}^{l}$      (4)

$X_{j}^{l}\left(\boldsymbol{X}^{l-1}\right)=\varphi\left(a_{j}^{l}\right)=\varphi\left(\sum_{i=1}^{N_{l-1}}\left(X_{i}^{l-1} \times W_{i j}^{l}\right)+b_{j}^{l}\right)$    (5)

where, $\left(X_{j}^{l}\right)$ represents the output of each neuron in the network at the $l$-th layer and the $j$-th index of the neuron within the layer, that obtains its inputs from all neurons in the earlier layer $\left(X_{i}^{l-1}\right), W_{i j}^{l}$ and $b_{j}^{l}$ are the neuron's weights and bias respectively, $\varphi$ indicates the neuron activation function.

Artificial Neural Networks (ANN) have been applied in several various fields such as computer vision, bioinformatics, natural language processing, and speech recognition, etc. Indeed, the CNNs is a type of artificial neural network, unlike the conventional neural network that cannot fully benefit from temporal or spatial features in the data.

In addition, the CNN network presents a new approach based on the spatially local connection and shared weights to integrate this information while considerably reducing the network complexity [33]. Therefore, the structure of the CNN network is essentially consisted of three main operations as follows: convolution, non-linearity and subsampling/pooling [34]. In short, CNNs can automatically extract and classify useful features from the input data using the operator of a convolution without the need for hand-learned features [35, 36]. As a result of which CNNs have gained a lot of interest in the visual field that includes images and videos.

Besides, the CNN algorithm is a feedforward network and it includes convolution layers, activation layers, and pooling layers. The principal functions of these layers are learning and extraction of robust features from the input data. Therefore, CNN is appropriate for addressing the problems of emotion detection from physiological signals and it is widely applied for learning and extracting optimal features from raw data, as well as classification tasks in different domains [37]. And to overcome the handcrafted features and to model the temporal and spatial structure in the ECG signals, 1D-CNN and 2D-CNN architectures are proposed in this research for ECG signals-based emotions classification.

4.6 The architecture of the proposed model

In this section, after performing the pre-processing steps of the ECG signals, the data are ready to be input into the PETSFCNN-GSO-DNN model, as depicted in Figure 8, which is based on deep learning (DL) techniques for classifying emotional states. Since the feature-learning process, which plays a significant role in classifying emotional states from ECG signals, is automated with CNNs, thus their use has become popular in this domain. Accordingly, the temporal and spatial features are extracted from ECG segments that have been normalized and reshaped with 2 parallel architectures, each of them containing convolutional, non-linearity, pooling, and dropout layers. Afterward, the outputs of both streams are first flattened, then fused using the concatenation method. Lastly, the GSO-DNN layer is followed to classify emotional states below.

4.6.1 Parallel-extraction of temporal and spatial features using convolutional neural network (PETSFCNN)

In this study, we proposed a novel end-to-end (PETSFCNN-GSO-DNN) model for detecting emotional states from ECG signals and classifying them according to the valence and arousal levels. The structure of the PETSFCNN-based feature extraction contains a parallel architecture of two types of CNNs are 1D-CNN for modeling temporal features and 2D-CNN for modeling spatial features. 1D-CNN is mainly used for modeling time-series or sequential data, which performs well in the extraction of their features [38]. For the above-mentioned reason, we have applied 1D-CNN for extracting temporal features. Besides, to extract more robust and useful features for classifying emotional states, we used 2D-CNN to extract spatial features from ECG signals after reshaping the 1-dimension ECG segments into a 2-dimensional gray-level map, as mentioned in part 3.2.4. The cause why we used 2D-CNN is that the two-dimensional convolution and pooling layers are convenient for filtering the spatial position of the 2D image signals. In extreme summary, convolutional neural networks extract deep features in both temporal and spatial dimensions of data, and thus, achieve a generalization performance for detecting emotions using physiological cues.

Finally, the architecture of the proposed model for extracting ECG features consists of two networks, as mentioned before. But for more information about the two networks, we have explained them in detail as follows. The first network is a 1D-CNN which is consisted of 2 convolutional blocks. The first block includes one convolutional layer composed of 32 kernels of size (1×5), followed by an activation function layer of type (Leaky ReLu), a max-pooling layer of size (1×2), and a dropout layer with a drop-rate (0.2). While the second block contains the same as the previous layers, except for the number of filters in the convolutional layer is increased to 64 filters of size (1×3), with adding a new flatten layer to it at the end. The 2D-CNN is the second network that has the same structure as the 1D-CNN network, but the only difference between them is, for example, that instead of being the kernel size is (1×m) will be (m×n). Both networks include the following layers:

(1) Convolution layer

Each 1-dimensional convolutional layer contains several feature maps, where each feature map consists of many neurons. Thus, the ECG signals in each convolutional layer are subjected to a 1-dimensional convolution filter/s to extract emotional features from those signals. Therefore, the extracted feature maps using 1-D convolution can be expressed as:

$F_{j}^{l}=\sum_{i=1}^{m}\left(X_{i}^{l-1} \bigotimes K_{i j}^{l}\right)+b_{j}^{l}$     (6)

where, $\otimes$ indicates the convolution operator, $F_{j}^{l}$ denotes the outputted feature map by $j$-th convolutional filter within the $l$-th layer, which obtains its input from the prior layer $\left(X_{i}^{l-1}\right)$, $K_{i j}^{l}$ and $b_{j}^{l}$ are the filter's weights and bias respectively, $m$ represents the size of the input vector in $X^{l-1}$. Likewise, the 2D convolution process can be calculated by:

$F_{i j}^{l}=\sum_{m=1}^{p} \sum_{n=1}^{p}\left(X_{(i+m, j+n)}^{l-1} \otimes K_{m n}^{l}\right)+b_{i j}^{l}$      (7)

where, each feature $F_{i j}^{l}$, acquired from multiplying $X_{(i+m, j+n)}^{l-1}$ which indicates the spatial location of the feature map at the previous layer $(l-1)$ with the $K_{m n}^{l}$ that represents the convolutional kernel/filter at layer $(l)$ with size $(p * p)$, then summed. Hence, the volume of feature maps can be calculated as defined Eq. (8):

Output $=\left(\frac{I-K+2 * \text { Padding }}{\text { Stride }}\right)+1$     (8)

where, I denotes the input size, K indicates the kernel/filter size.

(2) Activation function layer

The activation function increases or improves the network's nonlinearity [39]. And since the output of each convolutional layer is a linear computation process. Therefore, a non-linear activation is added to the linear action. Thus, the outputted feature map from convolutional layer is passed through an activation function of type leaky rectified linear unit (LeakyReLU). It is notable that the use of LeakyReLU activation after each convolutional output instead of using ReLU activation is for addressing the dying ReLU problem for negative values. The function of the LeakyReLU operation is displayed in Eq. (9).

$F_{j}^{l}=\varphi\left(X_{i}^{l-1}\right)=\max \left(\alpha * X_{i}^{l-1}, X_{i}^{l-1}\right)$,

$\max \left(\alpha * X_{i}^{l-1}, X_{i}^{l-1}\right)= \begin{cases}X_{i}^{l-1}, & \text { if } X_{i}^{l-1}>0 \\ \alpha * X_{i}^{l-1}, & \text { otherwise }\end{cases}$             (9)

where, $\varphi$ indicates the (LeakyReLU) activation function and $\alpha$ stands for a small value called alpha which was set to $(\alpha=0.01)$. It is necessary to note that applying a (LeakyReLU) function decreases the death of neurons in the network when the input $X^{l-1}<0$, which means that the output of LeakyReLU will have a small negative slope of $(\alpha=0.01)$ in contrast to the ReLU function the output will be zero. Figure 7 shows the difference between the Leaky ReLU and ReLU function [40].

7.png

Figure 7. Visualizing the LeakyReLU and ReLU functions

(3) Pooling/downsampling layer

Thereafter, the outputted features from the previous layer are passed into the pooling layer or so-called subsampling layer. The pooling layer performs downsampling based on local connection to decrease unnecessary data while keeping the beneficial information. The features generated by the max-pooling function can be defined as:

$F_{i j}^{l}=\max \left(f_{(i-1) 1+k,(j-1) 1+k}^{l-1}, \ldots, f_{k i,(j-1) 1+k}^{l-1}, \ldots\right.$,

$\left.f_{(i-1) 1+k, k j}^{l-1}, \ldots, f_{k i, k j}^{l-1}\right)$     (10)

where, $F^{l}$ is the output feature at layer 1 that derived from the previous layer $F^{l-1}$, here $\mathrm{k} \times \mathrm{k}$ denotes the spatial region size, and $(1 \leq i, j \leq(p-q+1) / k), \mathrm{p}$ represents the input size of feature map, and q refers to the kernel/filter size. Anyway, the max function simply decreases the input features within the spatial region based on the maximum value.

(4) Dropping layer

One of the most prominent obstacles in deep models is overfitting. And to increase the model’s generalization and regularization better, the dropout method is used. As some of the neurons randomly will be ignored during the training process according to a given ratio called dropout rate. Thus, the dropout operation [41] not only protects or prevents the model from over-fitting but also speeds up the model's performance. The dropout operation can be formulated as follows:

Dout $=\operatorname{dropout}\left(F_{i j}^{l}\right)$     (11)

4.6.2 Feature flattening strategy

As can be seen from the 1D-CNN network (Figure 8), the overall number of temporal features extracted from input data was 79872 ( $F^{T} \in \mathbb{R}^{79872}$ ). Whereas from the 2D-CNN network, the total number of spatial features extracted from input data was 14720 ( $F^{S} \in \mathbb{R}^{14720}$ ). Anyhow, the next step after extracting features is the flattening operation, which reshapes the different dimensions to have a 1-D shape. Therefore, the 2-D data must be flattened into a 1-D vector before being fused with the temporal features. For example, the flattening process can be expressed as:

$A_{i}^{S}=\left[\begin{array}{ccc}f_{11} & \cdots & f_{1 w} \\ \vdots & \ddots & \vdots \\ f_{h 1} & \cdots & f_{h w}\end{array}\right]=\left[\begin{array}{cc}1 & 2 \\ 3 & 4 \\ 5 & 6\end{array}\right]$     (12)

$F_{i}^{S}=\operatorname{flatten}\left(A_{i}^{S}\right)=\left[f_{11}, \ldots, f_{1 w}, \ldots, f_{21}, \ldots\right.$,

$\left.f_{2 w}, \ldots, f_{h 1}, \ldots, f_{h w}\right]=\left[\begin{array}{cccccc}1 & 2 & 3 & 4 & 5 & 6\end{array}\right]$     (13)

where, $A_{i}^{S}$ is a $2 \mathrm{D}$ spatial feature map of size hight $\times$ width; $f$ refers to the value of the feature. $F_{i}^{S}$ represents the flattened spatial features as a $1-\mathrm{D}$ vector.

4.6.3 Feature fusing strategy

After the temporal and spatial features are extracted and then flattened to vectors, they must be fused before being passed to the DNN model for detecting emotions. It is clear from Figure 8 and Table 1 that the feature fusion strategy is an intermediate level fusion in between the data level and decision level. It can prevent a large amount of computation and loss of information. Thus, to improve the performance of the proposed model for detecting and classifying emotions with high accuracy, the flattened features of both streams are fused using the concatenation strategy as described below:

$C F_{i}^{T} \in \mathbb{R}^{1 \times 94592}=\left(F_{i}^{T} \in \mathbb{R}^{1 \times 79872}\right) \oplus\left(F_{i}^{S} \in \mathbb{R}^{1 \times 14720}\right)$      (14)

where, $\bigoplus$ represents the concatenation operator, $C F_{i}^{T}$ refers to the total size of the concatenated features of both streams with a length of 94592 samples, $F_{i}^{T}$ is the size of the extracted temporal features with a length of 79872 samples, while $F_{i}^{S}$ denotes the size of the extracted spatial features with a length of 14720 samples. The model can get more inclusive and precise evaluation findings, by flattening and fusing the temporal and spatial features to a joint vector.

8.png

Figure 8. The architecture of the PETSFCNN-GSO-DNN model for detecting emotions

Table 1. The layer properties of the performed (PETSFCNN-GSO-DNN) model

9.png

Figure 9. Hyper-parameter optimization of DNN model using GSO algorithm

Table 2. The hyper-parameters of DNN classifier are tuned automatically using Grid Search Optimization (GSO)

Notes: LeakyRelu means a Leaky Rectified Linear Unit, Adam means an Adaptive Moment Estimation

4.6.4 GSO-DNN classifier

After extracting and fusing features process, the next stage is the classification process. Therefore, the final vector of the fused features is passed into the GSO-DNN model, which consists of two fully connected (FC) layers with 128 and 64 neurons each. Therefore, each fully connected layer is followed by an activation layer of type LeakyReLU, batch normalization layer, and dropout layer with a dropout rate of 0.2. Besides, to overcome and reduce the over-fitting problem, grid search optimization (GSO) algorithm is used for optimizing/tuning the hyper-parameters of the DNN model. The tuned parameters of the DNN model are presented in Table 2 and depicted in Figure 9. Lastly, the output layer of the DNN classifier consists of two neurons, where the first one represents low valence and the second one represents high valence (low arousal and high arousal), as showed in Figure 8 (classification layer). The activation function utilized with the output layer is the softmax function which can forecast the probability of the target class, as defined below:

$\operatorname{Softmax}\left(X_{j}\right)=\frac{e^{X_{j}}}{\sum_{i=1}^{n} e^{X_{j}}}, i=1,2, \ldots, n$     (15)

where, $X_{j}$  represents the input vector of the fused features, n indicates the number of emotional classes in the last layer. As the classification problem in this study is a binary classification task, therefore n=2. Furthermore, the loss function utilized in the output layer is based on the cross-entropy loss (CEL) function as defined in Eq. (16):

$C E L=-\sum_{i=1}^{C} L_{i} \times \log \left(X_{j}\right)$     (16)

where, $c$ is the number of emotional classes, $L_{i}$ indicates the (ground-truth label), and $X_{j}$ denotes the predicted label.

4.6.5 Implementation and evaluation metrics of the proposed method

For implementing the programming code of our experiment, we utilized the GPU of Google Colab via a laptop with the following specification: windows 10 Pro 64-bit, Intel(R) Core (TM) i7-4600U CPU @ 2.10GHz, 256-GB SSD, 12-GB DDR3 RAM. There are different methods utilized for reserving any data set as (training and test) data. This study suggests using the k-fold cross-validation strategy with k = 10 for dividing the dataset into training and testing sets. k-fold cross validation strategy is a resampling method used to reduce variance, bias, and errors in the dataset as well as aids to avoid overfitting problems and improves the performance of the model in terms of accuracy and stability. Therefore, 90% of the dataset was selected randomly for training the model and 10% for testing its performance (this operation is repeated ten times) [27]. The average of the ten test results is calculated to obtain the model's final efficiency. Besides, different metrics were used such as accuracy, F1-score, sensitivity, specificity, receiving operating characteristic (ROC) curve, and confusion matrix. All the above-mentioned metrics can be computed by four parameters in confusion matrix: TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative. The equations for these metrics are presented as follows:

$\operatorname{Accuracy}(\%)=\frac{(T P+T N)}{(T P+T N+F P+F N)} \times 100$     (17)

$\operatorname{recision}(\%)=\frac{T P}{T P+F P} \times 100$       (18)

$\operatorname{Sensitivity}(\%)=$ Recall $=\frac{T P}{T P+F N} \times 100$      (19)

Specificity $(\%)=\frac{T N}{T N+F P} \times 100$      (20)

$F 1-\operatorname{Score}(\%)=2 \times \frac{(\text { Recall }) \times(\text { Precision })}{(\text { Recall })+(\text { Precision })}$          (21)

$C . M=\left[\begin{array}{cc}T N & F p \\ F N & T P\end{array}\right]$       (22)