Evaluating the Impact of Emotions and Awareness on User Experience in Virtual Learning Environments for Sustainable Development Education

1.png

Figure 1. Merits of virtual learning environment

Vir tual Learning Environments (VLE) are emerging in the educational system. Hence, it is essential to comprehend how emotions and awareness influence students' perspectives and user experience in VLE. Therefore, we presented a novel Fuzzy-based Convolutional Neural Network (FCNN) for effective emotion evaluation. Figure 2 depicts the flow of the proposed method.

2.png

Figure 2. Flow of the proposed method

3.1 Data collection

This database contains facial images that were acquired for four separate sessions; the participants displayed a range of emotions while adopting a variety of head postures and exposure settings. There are almost 1,000 different images in all. at a resolution of 640 pixels by 640 pixels. In all, it covers the feelings that were conveyed by 300 participants who participated in any one of the lecture sessions. Smile, surprise, squint, contempt, and fury are some of the facial emotions that have been caught. The images were taken from a variety of tilt angles, and the quantity of light that hit the face was adjusted during the process [23].

3.2 Data preprocessing using histogram equalization

Assume that the input face pictures are made up of I distinct levels. The following Eq. (1) may be used to write the transforming mechanism.

$T_k=S\left(q_k\right), \quad k=0,1, \ldots \ldots \ldots, I-1$             (1)

resulting in level $T_k$ as the outcome of the source, normalized facial input level $q_k$. The transformation function $\mathrm{T}(\mathrm{r})$ for histogram equalization [1, 2] is defined by the connection.

$T_k=S\left(q_k\right)=\sum_{i=0}^k R_r\left(r_i\right)=\sum_{i=0}^k \frac{n_i}{n}, 0 \leq q_k \leq 1$ and $k=0,1, \ldots . I-1$           (2)

In Eq. (2), $n$ is the input level I pixel count and $R_r\left(r_i\right)$ is the probability density function (pdf) of the input picture. Consequently, the transforming function $T_k$ reflects the initial facial image's continuous dispersion function (cdf). This technique alters the contrast of a facial image by increasing the contrast of big items and decreasing the contrast of minor things, such as the background. While this outcome is suitable for generic pictures like images, it is less ideal if we desire a backdrop with a stark comparison. Figure 3 shows the procedure of histogram equalization [24, 25].

Histogram equalization may provide general brightness augmentation, despite the position in the input facial image. A sub-block is defined by histogram equalization, which also obtains its histogram data. Then, using the cdf of that sub-block, histogram equalization is carried out for the sub-center block's pixel. Up to the ending of the source image, the sub-block is then pushed forward by one pixel and the sub-block histogram equalization is reiterated. As a consequence of employing the nearby sub-block to histogram equalizes each pixel; the outcome is ideally suited to the local lighting conditions. As a result, the local contrast is maximized [26].

3.3 Feature extraction by using Discrete Cosine Transform (DCT)

Features are an image's representative measurements that set it apart from other data. The chosen attributes need to enhance the contrast between diverse images. By converting the image from the temporal domain to the spectral domain, features are retrieved. The image frequency describes variations in intensity or color throughout an image; it is not a consequence of time, but rather of spatial coordinates. To view content that is not evident in the temporal domain, the frequency data of the image is required. The characteristics are described briefly below. Energy is concentrated into lower-order coefficients through the discrete cosine transform (DCT). The DCT is entirely genuine. The DCT represents a series of infinitely many data elements as the sum of cosine functions that oscillate at various frequencies in an attempt to maintain the most crucial characteristics. The DCT coefficients for the converted output image, $A_{x y}$ in Eq. (3), are calculated using an input face image, $B_{n m}$, by the equation presented below. In the Eq. (4), A stands for the N-by-M-pixel face image, $A_{n m}$ for the intensity values in row $\mathrm{n}$ and column $\mathrm{m}$, and $\mathrm{A}_{\mathrm{xy}}$ for the DCT coefficients in a row a and column b of the DCT matrices. The feature extraction of a facial image using DCT is as follows in Algorithm 1. The process of DCT is depicted in Figure 4.

$A_{x y}=\alpha_a \alpha_b \sum_{n=0}^{N-1} \sum_{m=0}^{M-1} B_{n m} \cos \frac{\pi(2 n+1) p}{2 N} \cos \frac{\pi(2 m+1) q}{2 M}, \begin{aligned} & 0 \leq a \leq N-1 \\ & 0 \leq b \leq M-1\end{aligned}$            (3)

$\alpha_a=\left\{\begin{array}{c}1 / \sqrt{N, \quad a=0} \\ \sqrt{2 / N}, 1 \leq a \leq N-1\end{array} \alpha_a=\left\{\begin{array}{c}1 / \sqrt{M} \\ \sqrt{2 / M}, 1 \leq q \leq M-1\end{array}\right.\right.$            (4)

3.png

Figure 3. Procedure of Histogram equalization

4.png

Figure 4. Process of DCT

5.png

Figure 5. Architecture of FCNN

Algorithm 1: Discrete cosine transform

Gray-scale image blocks with a resolution of 640x640 pixels.

Using the Otsu approach, binarize the picture to produce text that represents binary 1 and backdrop binary 0.

Utilizing structural aperture, eliminate any nearby small stuff and perform the thinning procedure.

Apply DCT, split the resulting magnitude (facial image) into four equal, non-overlapping blocks, and then compute the standard deviations for the first and second blocks to retrieve the local patterns. That creates two characteristics.

To get approximation parameters (cA), vertical variables (cV), horizontal coefficients (cH), and diagonal coefficients, perform decomposition for the magnitude (image) of DCT (cD).

Calculate the standard deviation independently for the cA, cV, and cH frequency bands. This creates three characteristics and stores all the calculated characteristics in a vector.

3.4 Evaluation of emotion utilizing Fuzzy-based Convolutional Neural Network (FCNN)

The Oxford definitions describe the emotion as a powerful feeling that arises from one's surroundings, moods, or interactions with other people. It might also be described as a sensation made up of behavioral and psychological responses to interior and exterior situations. By analyzing facial expressions, machines can automatically discern emotions. Speaking, minor variations in facial expressions that convey various emotions may be utilized to communicate between individuals. The identification rate and the efficiency of facial expression identification have both been improved through the use development of several methodologies and approaches. CNN, which are artificial neural networks proficient in completing a variety of tasks including image categorization, is one of these methods. The essential premise of fuzzy logic is to substitute the set of real numbers {0, 1} with a member value in the range [0, 1]. Using a narrative phrase based on fuzzy logic with fuzzy definitions, systems based on fuzzy rules are utilized to construct a fuzzy inference system. The fuzzy information base, which comprises fuzzy If-Then rules, and a fuzzy system, that is utilized to apply the fuzzy logical procedure in the input to generate the output, make up the two essential parts of networks based on fuzzy rules. Hence, we proposed a combined novel technique Fuzzy-based Convolutional Neural Network (FCNN) to evaluate facial emotion. Convolutional, pooling, and fully connected layers make up the CNN's associated layers. The architecture of the FCNN is shown in Figure 5.

The Convolutional layer is in charge of utilizing a convolution operation with filters to extract a characteristic mapping from the initial facial image. Sliding the filter over the initial facial image is how the convolution procedure is carried out. Everywhere, a matrix multiplication procedure is carried out, and the output is added to create the feature map. A stride value regulates the convolution around the input space and governs the filter's moving step. The filter typically moves pixels at a time. This model's primary job is to categorize the student's facial expressions using the images that the facial detection system provides. The method also offers the n expression categories' possibility levels. We choose the database that will be used to train and evaluate the FCNN model before building it. During virtual learning, we choose the students' face images. The model is trained on 80% of the images that we choose, with a balance of 20% being used for testing. It has three different kinds of layers: fully connected, pooling, and Convolutional. The suggested CNN model's training hyper-parameters are shown. The number of epochs, batch size, and the learning rate is used for training. Much conventional feature recognition employ convolution, a face recognition method that is focused on the idea of a responsive field. A Convolutional kernels mask of the source matrix's sliding window is used during convolution and is calculated using the following Eq. (5).

$E_{i j}=\sum_{i=0}^{K_a} K \sum_{j=0}^{K_b} y_{i j} \times k a_{i j}$                (5)

where, $K_a$ and $K_b$ are the Convolutional kernel's size and length, respectively, and $E_{i j}$ is the output matrices. To preserve the identical size and length of the matrices after convolution, the border of the intake matrix is often padded with 0 , and 0 is still 0 after the Convolutional process. Typically, $K_a=K_b$ is set as a square kernel, $y_{i j}$ is the input matrix, and $k a_{i j}$ is the weight of the convolution kernel. Thus, padding 0 won't have an impact on convolution. The activation function is used to provide nonlinear outcomes for networks that have been uniformly coupled. However, the sigmoid, which was utilized in previous networks, emerges and the gradients vanish over reverse dispersion when the network is large and the number of layers is huge. As a result, FCNN uses ReLU as its activation function. The following Eq. (6) represents the definition of ReLU.

$g(y)=\max (0, y)$              (6)

The input facial image is condensed after the convolution procedure to reduce the intricacy of the network processing. Using pooling, the dimension may be decreased. The input matrices are used in pooling, but a sliding window mask is used; the masks do not intersect as it moves. In addition, to allow an NxN mask to minimize 1/N2 instances of the source feature matrices and hence lower the dimension, the step must be equal to the height of the Convolutional network. The two types of pooling are typically maximal and average pooling. In this case, the output is the greatest value from the mask, with all other values being immediately eliminated. Optimal pooling is employed for the Convolutional process since it is often preferable to keep the most significant features and theoretically, less significant features shouldn't alter features. Average pooling employs the outcome layer since it determines the aggregate of all elements in the mask. A huge target might receive a high result during the calculation process due to average pooling, which has a physical significance in emotion recognition. Traditional fuzzy integrals' fuzzy density is often adjusted by user feedback. The formula Y=yi = 1 states that m represents a set of m classifiers or the number of CNNs utilized in this research, and g(yi) denotes the fuzzy measure, which is taken into account while determining the value of the subset xi. In other words, yi is the result of the ith classifier, and g(yi) indicates the level of confidence for this class. If the fuzzy measure value is 1, it means that this output may be completely trusted. The fuzzy measure ranges from [0, 1]. If the value is 0, there is no reference value for the output. These fuzzy measure characteristics are shown.

• When all classifier outputs are consistent, as they are when g(Y) = 1, the findings may be believed.
• All classifier outputs are not taken into account when g() = 0.
• The function of a fuzzy measure should be incrementally monotonic is represented as in Eq. (7).

$X \subset Y \subset Z$, then $0 \leq g(X) \leq g(Y) \leq 1$          (7)

Starting with a fuzzy density that only has one element in g(A), such as g(a1) and g(a2), will allow one to get the fuzzy measure. It is shown how fuzzy measurements and fuzzy density relate to one another. Fuzzy density is defined as g(a1, a2, and a3), where g(a1, a2, and ya3) is the certainty level of the particular class created by merging three CNNs.

3.5 Evaluate awareness by questionnaire survey

In a VLE, awareness evaluation is vitally important for enhancing both instructors' and students' user experiences. A questionnaire survey approach could be employed to increase awareness. To measure awareness, several questions are presented to both students and instructors and their responses were validated. Table 1 shows the measure awareness, several questions and the questions as below:

(1). Is a virtual learning environment beneficial to education?

(2). Has the implementation of VLE in school education improved your skills?

(3). Should the VLE's user interface be changed for e-learning?

(4). Whether instructors and students can quickly get interaction when using a VLE?

(5). Do you find it difficult to comprehend concepts in a virtual learning environment?

The interface of a virtual learning environment may be customized, according to this survey. Implementing advanced applications in a virtual learning environment could significantly facilitate interactions between students and teachers.

Table 1. the measure awareness

Yadav [17] presented a support vector machine (SVM) for facial emotion identification and the ability to recognize seven various outward appearances of two individuals, such as anger, sorrowful, pleased, disgusted, indifferent, fearful, and surprised expressions in the dataset. When there is a large data collection, it does not operate very effectively since the necessary training period takes a longer amount of time. The study by Ackerson et al. [18] recommended using a multimodal approach and recurrent neural network (MA-RNN) to emotion detection, which integrates many different modes into the RNN architecture in an attempt to increase the accuracy of emotion detection. When the gradient is too low, the changes to the parameters are rendered meaningless and unimportant. Because of this, the processing of lengthy data patterns is made more complicated. Hu et al. [19] developed an approach known as joint adaptive fine-tuning, or JAFT, to combine localized and global information in addition to enhancing the effectiveness of facial emotion classification. This technique can recursively alter the weights of the network. There is a correlation between lowering the dimension of weight in the network and lowering its accuracy. Zhang et al. [20] suggested spatial and spectral graph neural network (SSGNN), is intended to retrieve spatial and spectral context characteristics from face images in recognizing emotions based on both macro- and micro-expressions. Efficiency suffers if there is less processing of graph edges according to the properties and linkages they contain.

The dataset used for training and testing the FCNN model was selected based on the availability of publicly available datasets that contain facial images with labeled emotions. The dataset used in this study is the Japanese Female Facial Expression (JAFFE) dataset, which contains 213 grayscale images of Japanese female facial expressions. The criteria for choosing emotions to capture were based on the six basic emotions (happiness, sadness, anger, fear, surprise, and disgust) and the neutral expression. However, it would be beneficial to provide more details on the diversity of participants in terms of demographics, such as age, gender, and ethnicity. DCT is chosen as the feature extraction technique because it is a widely used method for image compression and has been shown to be effective in capturing facial features for emotion recognition. DCT is relevant to emotion recognition from facial images because it can capture the spatial frequency information of the image, which is important for identifying facial expressions. The algorithm for DCT is provided in the paper, and it involves dividing the image into non-overlapping blocks and applying the DCT to each block to obtain the frequency coefficients. The training process for the FCNN model involved dividing the dataset into training and testing sets, with a ratio of 80:20. The model was trained using the Adam optimizer with a learning rate of 0.001, a batch size of 32, and a total of 100 epochs. The fusion of features from different frequency bands is achieved by calculating the standard deviation independently for the cA, cV, and cH frequency bands and storing all the calculated characteristics in a vector. While the paper does not provide a comparison of the proposed FCNN method with other existing methods, it does evaluate the performance of the FCNN method based on accuracy, sensitivity, specificity, and precision.