Detecting Student Engagement with Convolution Neural Network and Facial Expression Recognition

For e-learning systems, predicting students' engagement in a class is essential [1]. The tool enables the instructor to observe how the student interacts with numerous aspects. The learning outcomes will increase with the depth and duration of the student's engagement in the lesson. Research in educational theory has always focused heavily on student engagement [2]. Early attention to engagement was prompted by concerns about high dropout rates and data showing that many students, between 30% and 70%, reported feeling bored and disengaged in the classroom regularly. The amount of student interest or disengagement might be determined using eye movements [3], facial expressions [4], gaze patterns, eye tracking, and body movements [5]. An outside observer can evaluate how engaged people are.

Analyzing student engagement could help accurately measure the mentioned characteristics. The advancement of student engagement is helpful for online learning and teaching in conventional classrooms, innovative and challenging educational games, and online tutoring platforms [6, 7]. Any learning environment may be considered a combination of cognitive, behavioral, and emotional states regarding student engagement. Active engagement and participation of the student in class activities are necessary for behavioral engagement to occur. On the contrary, behavioral engagement involves students' grasping information and self-dedication to the learning process.

In contrast, emotional engagement indicates the student's affective state and direct engagement in the class [8-10]. Emotionally and intellectually engaged students are more likely to work more, persevere longer, and meet course requirements more successfully than less involved students. As a result, this component is essential in defining each student's welfare and growth. Emotional engagement in interactions with instructors and peers indirectly modifies children's increased cognitive involvement [10]. The level of student engagement affects upper-body posture, facial expressions, and general ecological factors. However, in an online learning environment, facial expressions are the most intuitive behavioral way to demonstrate interest [11]. For the most part, datasets included the seven primary emotions of grief, pleasure, neutrality, surprise, disgust, wrath, and contempt [11-13]. Now, the emphasis is on monitoring students' emotions during learning to gather particular information about their engagement. Recent research has linked facial expressions to self-reported and assessed emotional conditions related to learning, including boredom, engagement, confusion, and frustration [12]. In a synchronous learning scenario, it is important to consider the connection between the intensity and timing of facial expressions [13].

ML and Deep Learning (DL) are used in two approaches to measure affective expression. Using hand-drawn patterns, the ML-based technique gathers facial data to determine engagement levels [14]. In emotional state prediction tests, DL techniques outperform traditional ML-based methods because they learn features from training data, permitting the algorithm to recognize perfectly all right facial differences [14, 15]. Additionally, DL-based systems are non-intrusive. The tools they utilize to collect and analyze facial expression video data are accessible, controlled, and straightforward. The two types of DL techniques are now used for emotion identification: Video sequence-based techniques and approaches based on static images [15]. Classifying facial emotions from a sequence of linked frames in a video is more natural because a video sequence provides FER with substantially more information than a static image of a person's face. There are three different types of video sequence-based approaches: 3D CNN, which is a combination of CNN and Long Short-Term Memory networks, as well as a combination of spatial and temporal CNN models [16, 17].

Researchers created a DL model comprising two critical steps to overcome the abovementioned issues: fundamental facial expression identification and student engagement recognition. The FER-2013 dataset was initially used to train a CNN for a facial representation model and achieve good performance. A student engagement recognition model was created using a separate CNN and trained on a compiled student engagement recognition domain dataset. Then, we employed the model to activate it. Figure 1 illustrates the conceptual framework for student involvement in an e-learning system that uses the DL technique. It begins by sending a dataset named FER2013 to the place where the data is being collected for further analysis or processing. On the other hand, students' engagement in learning activities starts with online learning. The data is then pre-processed before being passed to a deep-learning classifier like CNN or LSTM. Pupils are either engaged or disengaged when they accept a learning approach.

3.1 FER dataset

The most used dataset for applications using FER. Over 35,000 grayscale photos of faces with each classified with one of seven distinct emotions, make up the dataset: Positive: joy and surprise; negative: rage, disgust, fear, motivation, and sadness; and valence: neutral [15-17]. A label identifying the facial expression shown in each image serves as a classification. The challenge of identifying facial expressions in images is often evaluated using the FER dataset to test the effectiveness of ML algorithms.

One of the strengths of the FER dataset is its large size and diversity. The dataset includes images of people of different ages, genders, and ethnicities and images captured under different lighting conditions and with varying head poses and facial expressions. This diversity makes the FER dataset valuable for FER in real-world settings where faces appear in varied contexts and conditions. Another strength of the FER dataset is its availability and ease of use. The dataset is publicly available, which makes it accessible to researchers and developers around the world. The FER dataset comes with a standard training and testing split. This makes it easy to compare the performance of different FER models on the same set of images. This adjustment helps ensure that the results of different studies can be compared and combined, leading to more rapid progress in FER.

3.2. Data description

The FER dataset is a standard benchmark for assessing the effectiveness of machine learning models in FER tasks. It generally comprises 35,000–40,000 images. The photos in the dataset were gathered from people of different ages, nationalities, and genders to provide a varied and representative collection. Usually grayscale, the images in the collection are scaled to 48×48 pixels. The details of the FER dataset are shown in Table 2.

Table 2. FER dataset

The list of emotion categories with the number of images in the FER dataset is given in Table 3.

Table 3. The number of images of each emotion in the FER dataset

The following is a breakdown of the FER dataset into its training and test sets.

3.2.1 Train dataset

Each emotion image is organized into seven categories in the train folders. The train data distribution is shown in Table 4.

Table 4. Train dataset

3.2.2 Test dataset

There are seven folders for each emotion image in the test folders. The distribution of test data is displayed in Table 5 below:

Table 5. Test dataset

2.png

Figure 2. Distribution of emotions in a training dataset

3.png

Figure 3. Distribution of emotions in a testing dataset

3.2.3 Data visualization

The train and test folder's subfolders each include emotional image files. Data is spread unevenly, and as a result, some emotions will be identified exceptionally effectively. In contrast, others will be caught very poorly. The training and testing folders are seen in Figures 2 and 3, respectively.

The sad emotion includes more images than the other emotions in the training and testing folders, which is evident from the above visualization and suggests that each emotion is evaluated differently. Compared to the other emotions, disgust includes fewer images, indicating that there would be relatively few data points from the disgusting class during training, resulting in lousy evaluation metrics outcomes.

3.2.4 Images visualization

According to Figures 4 and 5, each folder contains several images for each mood. To further illustrate the variety of images we have, we separated each class image from the training and test files into its graphic.

The following visualization demonstrates that virtually all images are identical in size and greyscale. In the next stage, we will prepare the training and testing datasets to guarantee that all images have the exact attributes.

Figure 4. Training emotions with labels class

5.png

Figure 5. Testing emotion with label class

Using 30 epochs, the model in the previous section trains the model on the training dataset "training_set" and verifies it on the test dataset "test_set."

An instance of the "Sequential" class from the TensorFlow "tf. Keras" API makes up the "model" object, which has a method called "fit" that does fit. In this example, "training_set," the "fit" function trains the model on the provided training data.

When a model is trained, a validation dataset is used to assess its performance, and this parameter provides the validation dataset. The model's performance on this validation dataset can be employed to track the training status and avoid overfitting.

The number of "epochs" arguments defines how often the training data will be used to train the model. The model analyses the training dataset once every epoch, changing its weights and biases to reduce the loss function.

The "history" object stores the results of the "fit" function, which include details about the training procedure and the model's accuracy and loss on training and validation datasets over time.

Table 6. Hyperparameters of the proposed model

As shown in Table 6, hyperparameters are settings made before a neural network's training process rather than ones learned as the network is trained. Among them are:

Optimizer: Depending on training data, network weights may be iteratively updated using Adam, an optimization technique. Adagrad and RMSprop are two more optimization methods combined for maximum benefit. It updates the network weights using the gradient of the loss function. With the learning rate lr set to 0.0001 and the decay set to 1e-6, the model's optimizer is in this code throughout the training phase.

Epochs: The model will be trained the entire training dataset a finite number of times called iterations or epochs. A training set period is a set of forward and backward passes. After each epoch, the model corrects its weight based on the difference between predicted and outgoing output. Because this code has 25 epochs, the model will be trained on the training set 25 times.

Loss: The loss function, sometimes referred to as the cost function, measures how off the given model is as a prediction. The optimization process then applies the loss value to adjust the model’s weights. For this code, the categorical cross entropy loss function is used, which is employed when the problem is multi class classification where the result is in the form of probability distribution cross the class.

5.1 Model summary

To determine student involvement using the FER pictures dataset, this model builds a CNN model in TensorFlow using the Keras API. Multiple layers, including MaxPooling2D, Conv2D layers, Dropout, BatchNormalization, Flatten, and Dense, are included in the model. Conv2D layers collect features from the picture, down-sample it using MaxPooling2D, regularise it with Dropout, flatten it by converting it to a 1D vector, forecast the final output using Dense layers, and regularize the activations of the preceding layer using BatchNormalization. Accuracy, categorical cross-entropy loss, and the Adam optimizer are the measures used in the model's construction. For 30 iterations, the model is trained using the training set. A history of the model's accuracy and loss is kept throughout training and authentication.

Indeed, it would be worthwhile to perform a detailed comparison of the suggested CNN architecture to existing solutions for student engagement detection, and we are thankful to the reviewer for pointing this out. In this response, based on the presented model, we describe its peculiarities and reveal certain features to improve its work. Moreover, we report a comparative table as a breakdown of how our proposed architecture contrasts with the state-of-the-art in the field.

This kind of architecture can be considered a unique and isolated feature of our CNN architecture.

5.1.1 Layer configuration

The proposed model consists of a layered structure. In the first stage, Conv2D layers are used with filters 32 and 64 since such structures identify spatial hierarchies in the input image. Such a sequential structure proves effective in learning the combination of simple and complicated features of the network.

5.1.2 Batch normalization

The addition of a BatchNormalization layer after the convolutional layers speeds up the training process by normalizing the layer's input, a factor essential in superdeep networks.

5.1.3 Dropout layers

To reduce overfitting, we added two Dropout layers with probabilities of Dropout of 0.25 and 0.5, respectively. This regularization technique becomes critical given the relatively small variation in the FER dataset, therefore improving its ability to generalize to unseen data.

5.1.4 Dense architecture

The architecture reaches a fully connected layer of 1024 neurons and a softmax activation function in the last layer of the network. This high density helps aggregate the features learned through convolutions and pooling to enhance the improvements in the engagement level classifications.

5.1.5 Optimization strategy

We use the Adam optimizer with a learning rate set to 0.0001 and decay at 1e-6, which has been observed to work well for image classification tasks. This fine-tuning is essential as it will help us achieve the right convergence while training.

5.1.6 Comparative analysis

This section includes a comparison of findings with state-of-the-art models.

Table 7 presents a comparison between our proposed model and other prominent CNN architectures used for FER and engagement detection.

Table 7. Pretrained model comparison

(2) Computational efficiency: The model reached roughly 1.5 billion FLOPS, allowing it to work in real world scenarios like detecting engagement in very different environments e.g., from classrooms. Our CNN architecture Joint Feature Learning and Sparse Representation based on Experimental Results demonstrates such distinctive features in order to improve detection performance while retaining the efficiency needed, for practical application in education. Our model is further compared to other architectures of similar complexity in estimating student engagement with a comparative table. The points presented here will be discussed in detail in the revised manuscript in a more detailed way upon submission to make it clearer.

5.1.7 Real-time engagement recognition and interpretation

Computational efficiency: As a next step, we will give an extensive description of computational features needed to train the CNN model and necessary computations for the inference stage. Key metrics to be discussed include:

Inference Time: The time the model takes to analyze an input frame and give the corresponding output prediction. This metric is vital for real-time engagements due to the real-time effect, where responsiveness to the engagement detection directly affects the stated metric.

Training Time: How long it took to train on the dataset, the number of iterations, and the hardware used. This will help understand the feasibility of the proposed model of engagement detection in the context of educational institutions interested in implementing engagement detection systems.

Memory Usage: The distribution of RAM and GPU memory used during inference. This information is crucial when implementing the model on devices with reduced functionality, such as tablets or low-end laptops, which are characteristic of learning institutions.

Pre-processing Time is the time spent getting data and pre-processing it. Here, we have used video frame extraction, normalization, and augmentation.

The main computational performance metrics are outlined in this part of the paper.

We devoted a section and Table 8 that outlines our computational analysis regarding efficiency, latency, and training costs.

CPU and GPU usage: The training process of our convolution neural network model on Google Colab will use CPU and GPU resources, which consume energy and have an environmental impact.

Several factors need to be considered to estimate my project's computational cost. These include the laptop's energy consumption, the machine learning model's memory usage, and the data transmission and storage.

Table 8. Computation cost

Since my laptop specifications are not optimized for high-performance computing, my machine learning project's energy consumption would be lower than using specialized hardware. However, the training process of my convolutional neural network model would still consume a significant amount of energy and have an environmental impact.

Additionally, since the training dataset size is 112 MB and the testing dataset size is 28 MB, the machine learning model's memory usage would depend on the model architecture and batch size used.

During the training process, approximately 20 experiments were run to reach a training accuracy of 94% and a validation accuracy of 93%. The training time for one epoch was 1 hour and 136 seconds per step, and 57 steps were taken. The total computational cost is estimated to be several hundred kilowatt-hours (kWh) of energy, which is equivalent to several hundred kilograms of carbon emissions.

In conclusion, my machine learning project would have a relatively low computational cost compared to specialized hardware. However, it would still significantly impact energy consumption and the environment. To promote durability in machine learning research, it's important to explore ways to optimize the machine learning models to reduce their energy consumption, such as using more efficient algorithms and reducing the size of the datasets.

By including these analyses in our paper, we hope to offer a broad view of how our engagement detection model works in practice. Such specific elaboration of computational complexity, possible cost of training, and potential limitations will not only help to demonstrate the possibility of applying our model to a real educational environment but also help educators and institutions make wise decisions about its applicability.

5.2 Model evaluation

The process of assessing a model's performance on a given dataset is called model assessment in machine learning. The evaluation aids in determining the model's success in resolving the issue it was created to address. Precision, accuracy, F1 score, recall, and other metrics can be used to assess the model's performance. For example, the accuracy metric gauges the model's proportion of accurate predictions. It assesses the CNN model for FER dataset student attention detection. Utilizing techniques like the confusion matrix and ROC curve, the assessment is carried out on a different test set and can be examined.

The following are the evaluation metrics considered in this model:

Precision: The behavior of optimistic forecasts compared with all entitle positive forecasts, in percentage.

Accuracy: The percentage of samples that where categorized correctly.

F1 score: The mean of the accuracy and stand of recall is called the F1 score.

AUC-ROC Curve: It represents the area under the curve of receiver operating characteristic; abbreviated as AUC-ROC Curve.

Recall: The ratio or proportion of the positive samples that where correctly categorized as positive.

Confusion matrix: Outlook of FP, TP, FN, TN is presented. There is no significant difference between this classification model's training and validation accuracy, preventing overfitting or underfitting. Instead, the training accuracy is 94%, and the validation accuracy is 92%, demonstrating that the model was correctly trained and matched the validation data perfectly.

Another popular set of metrics which are used for a classification model to assess its ability to predict the target classes on the training dataset is training accuracy. The relative error in testing could also provide an understanding of how well the patterns in training data have been adopted by a particular model since it may be used for tracking of over-fitting or under-fitting.

A model should have a high training accuracy because this means that the model is capable of correctly placing the training data into groups. But high training accuracy sometimes results in the model that fits the unlabelled data well. Therefore, it is essential to use the data from a validation or a test set to evaluate the model. This is caused by modeling that retains all the details of the training data set and as a result the computer learns a lot of irrelevant data which yields high training accuracy but low testing accuracy because the model is too complex.

From Figures 9 and 10, it is clearly shown that the model has high training accuracy and nearly perfect validation accuracy, this imply that the model performs well on unseen data.

9.png

Figure 9. Training accuracy performance

10.png

Figure 10. Training/validation loss performance

A classification model's "training loss" is the difference between goal values for a specific training dataset and expected outcomes. To minimize training loss and ensure that the model's predictions are as detailed as is practical, it serves as an optimization objective during training. A minor training loss means the model fits the training data more closely. In contrast, a more significant training loss means the model has trouble seeing patterns in the data. The identification of underfitting and overfitting can be aided. Underfitting refers to a model that is too basic and fails to recognize patterns in the data. In contrast, overfitting is a highly complicated model for remembering the training data.

Moreover, the F1 score is particularly important when there is controversy between its level and a number of precise points. Moreover, its usage is preferred when the dataset is unbalanced in that it takes into consideration both false positive and FN.

To review, accuracy assesses the overall correctness of the predictions and generalizes to average precision, which assesses the proportion of accurately predicted positive options, and recall, which describes the proportion of actual positive correlations that are truly recognised among them.

From Table 9, TN is the percentage of negative samples recognized by the model.

FP is the proportion of negative samples that the model regards as positive.

TP is the number of positive samples that the model identified as positive.

FN represents the proportion of positive samples that the model identified as negative.

The confusion matrix offers a method for examining the model's performance and may be used to identify areas needing improvement. For instance, as shown in Table 10, a large number of FNs can suggest that the model needs to be enhanced to better identify positive samples.

The confusion matrix evaluates model performance. It also highlights areas for improvement. For instance, a high false negative rate may signal that the model needs to enhance positive data recognition.

The suggested model's confusion matrix for all seven emotions is shown in Table 11. The model's few FNs indicate good performance.

Table 9. Evaluation metrics of the proposed model

Table 10. Confusion matrix with a large number of FN

Figure 12. Database sample images

5.6.1 The outcome of the integration of the UPNA head pose database

To test the feasibility of integrating the UPNA Head Pose Database into the model training, we conducted experiments to compare the levels of engagement detection with the FER dataset using only the combined dataset containing the UPNA data. Table 14 shows the model performance on the UPDA database.

Improved accuracy: The integration of the UPNA Head Pose Database improved the overall performance from 90.2% using only FER to 92.1% using the combined data set, showing that the model had improved its ability to identify emotions for different head poses.

Higher F1-score: Thanks to all the experiments, we showed an increase from 0.911 (from FER) to 0.933 (combined dataset). This means a reduced trade-off between recall and part, which is helpful when identifying engagement signals that are often hard to observe across a range of class settings.

Robustness against variability: The combined dataset enabled more consistent model performance under different lighting conditions and head poses for all angles since the number of false negatives was reduced from 22% for the FER only to 15% for the combined dataset. Thus, the actual level of students' engagement is better captured.

Table 14. Model performance on UPNA database

Training time and computational cost: Although including such information doubles training time by approximately three hours, the enhancement in performance makes it worth it. This training was conducted on a GPU, which could handle the large dataset in this improved model within a comparatively short inference time of 120 milliseconds per image.

5.6.2 Dataset bias and limitations

In this paper, we have included a discussion on the biases introduced by using only the FER dataset, explicitly highlighting the following:

Emotional expression variability: How can students' affective states differ in a naturalistic learning environment as opposed to benchmarks assembled in a database?

recommendations for future research: Therefore, we propose that subsequent research consider capturing actual engagement data to improve the model's generalization so that our detection systems can run in other learning environments.

Therefore, augmentation of the UPNA Head Pose Database in the proposed model has been proven to provide an enhanced capability of engagement detection. This helps resolve the drawback of solely using the FER data set and opens up better prospects for accurate determination in actual use.

5.7 Multi-modal integration of FER and eye-tracking

In connection with what has been said about integrating facial emotion recognition (FER) with other techniques mentioned, including eye tracking, it is understood that more extensive dialogue is needed on how these three complementary methods can be combined most effectively. In the following section, we describe changes we will implement in the paper to address this aspect of multi-modal integration for better engagement recognition.

Overview of Multi-modal Approaches: In the first article, we will describe the benefits of integrating FER with eye tracking and other methods for identification of engagement. This section will cover how these technologies can complement each other by providing different perspectives on student engagement:

FER helps one understand the emotional status of students, which can inform one whether the students are engaged or disengaged during learning activities.

Eye Tracking Yields knowledge about students' locations and can be used to better understand their orientation to the lesson content or to other students and the instructor.

5.7.1 Integration framework

Data fusion techniques: It includes different approaches to combining FER and eye-tracking data.

Early fusion: When feature extraction is complete, two modalities of raw data are merged into a single input feature vector and sent for further analysis in FER and eye-tracking processes.

Late fusion: Using different models for FER and eye tracking data, each of which is applied based on a separate set of pathways, later aggregating the outputs (for example, prediction results or probabilities) through practices that include averaging, weighted voting, or stacking.

Hybrid fusion: One category is a hybrid of early and late fusion, meaning that features are fused initially and then transformed by separate models.

5.7.2 Proposed architecture for integration

We will describe how FER and eye-tracking systems can be combined into a unified structure. The same is shown in Table 15. This may involve:

Table 15. FER, eye-tracking, and multi-modal values

Shared feature extraction layer: An LSTM with an additional convolutional stack used for FER that takes in video frame inputs and passes both through the network, including gaze direction and attention parse from eye-tracking input.

Joint decision layer: A model that integrates outputs from the FER model and the eye tracking model and makes one final decision on the level of engagement among the students.

5.8 Comparative analysis

Table 16 displays a comparison chart for the study on identifying facial expressions of emotion. Various parameters may be used to compare the performance of models produced in previous articles. These include the model's architecture, the amount of the dataset used, the type of data pre-processing used, the optimization strategy used, and the training methodology.

Since we used a more robust optimization algorithm, a larger, more varied dataset, and a more refined architecture than the other models, we can compare our model to other methods and find that the CNN model obtained 94% accuracy, which is higher than the other models. In addition, the CNN model performed better than other classifiers due partly to the training technique, which boosted the model's performance more effectively.

According to the study articles and model accuracy, our model performs better than the other models. Although it is 5% less accurate, the model that utilizes the FER dataset and has an accuracy of 89% [35] performs similarly to ours. In comparison, the accuracy of the models with 57% [31] and 72% [36] is much lower. The complexity of the model, data calibration, hyperparameter tuning, and assessment strategy are only a few factors that might influence a model's accuracy.