A Dual-Layer, Content-Aware Framework to Validate Online Student Engagement via ML-Based Comprehension Assessment

The sphere of artificial intelligence-based surveillance of student engagement has been evolving quickly along with the popularity of online studies. The first systems mainly used visual cues (eye gaze and facial expressions and the position of a head) to determine the presence of attention during virtual meetings. Hossen and Uddin [1] used models that were based on convolutional neural networks to identify attentiveness using webcam feeds, despite the fact that they tended to have low accuracy in uncontrolled settings. There were also commercial visual engagement analytics at scale tools like GoGuardian [2] which provided attentiveness dashboards on live lectures in real-time. However, such systems mostly focus on behavioural cues at the surface level and cannot detect if learners are processing information or retaining whatever they are receiving.

The restrictions of engagement tools, which are based on the visual input only, are widely documented in existing literature. According to Ma and Shi [6], there had been a continued discrepancy between objective performance of students and how students self-rated their attentiveness in virtual classroom settings. Wang and Zheng [5] made an argument in a parallel study that visual cues are not reliable proxies of actual learning outcomes, particularly in cases where learners are looking at each other but failing to process learning material. All these observations echo the larger concerns about the legitimacy of web-based analytics, especially when such environmental factors as distractions at work, unsuitable lighting conditions, and privacy limitations undermine fidelity of data [9]. As a result, even the largest videoconferencing tools, such as Zoom, have rolled out their attention-tracking tools due to the reliability and ethical issues associated with them [4].

Multimodal interaction identification Multimodal interaction detection in order to overcome these shortcomings, recent research has sought to adopt multimodal interaction detection, combining visual, audio, and affective techniques to enhance predictive power. The evidence presented by Angeline and Nithya [12] indicates that multimodal fusion leads to increased engagement detection accuracy; however, the literature available does not include much information on the ways to determine whether the understanding or actual learning is achieved. Simultaneously, progress in natural language processing, particularly transformer-based models, has led to a variety of novel tasks, such as automated summarization and automatic question generation. Zhu et al. [14] used multimodal transformer to derive brief summaries of educational content to teach people, but Dhanya et al. [15] used the T5 model to get multiple-choice questions based on the lecture transcripts.

Despite these technological advances, a critical gap remains:

(1) Engagement monitoring systems are rarely cross-validated against actual student understanding.

(2) Most tools continue to assume that attention equates to comprehension, leading to potential misclassifications. For instance, students who appear disengaged due to camera issues or personal habits may score highly on assessments, while others who maintain screen focus may perform poorly.

This highlights the need for systems capable of verifying engagement claims through performance-based validation.

Novelty relative to prior work: Prior studies primarily (i) estimate engagement from visual behavior, (ii) combine multimodal signals (e.g., audio, context), or (iii) assess learning outcomes in isolation. To the best of our knowledge, no prior classroom study has positioned content-aligned comprehension as a systematic validation criterion for vision-based engagement dashboards in live instruction. The proposed framework operationalizes this link and reports when and why behavioral and cognitive indicators diverge. This work makes three contributions:

(1) Introduced a content-aware comprehension layer that validates commercial behavioral E-scores using immediate post-lecture C-scores.

(2) Implements a Whisper–BERTSUM–T5 workflow, optimized for GPU acceleration yet deployable on a single workstation in classroom settings.

(3) Across ten lectures involving one hundred students, demonstrates a weak E–C correlation and provides mismatch analyses showing when behavioral attention diverges from learning comprehension, thereby informing pedagogical feedback.

Table 1. Comparison of existing engagement monitoring methods

Table 1 summarizes the key characteristics of two widely adopted engagement monitoring methods: Traditional Visual Monitoring and Multimodal Systems. Earlier engagement systems mainly relied on what they could see — a student’s face or eye movement — to judge attentiveness. These methods are easy to scale and need little hardware, but they fail to show how much the learner actually understands and often give mixed results when lighting or background changes. Some newer models combine different types of data, such as sound or body signals, to make the predictions more stable. However, these systems need additional sensors and are difficult to apply in large online classes. More importantly, both approaches overlook what the student actually understands during the lesson, showing why a validation layer like the one proposed in this study is necessary.

This section explains the setup, data, and steps used to test the dual-layer engagement validation framework in real classroom sessions. The configuration was designed to replicate realistic online-learning scenarios where computer-vision-based engagement monitoring runs in parallel with automated comprehension assessment.

4.1 Experimental environment

The experiments were conducted in a controlled virtual-classroom setup integrating a standard video-conferencing platform (Google Meet) with a commercial vision-based engagement monitor (GoGuardian) [2]. Ten independent 45-minute lectures were delivered to a cohort of 100 undergraduate students from the Computer Science programme. During each live session, facial-orientation, gaze-direction, and posture data were continuously analyzed to compute real-time engagement scores (E-scores) ranging from 0 to 100.

Table 2. Reviews the components employed in the study

In parallel, lecture audio was routed to the Whisper automatic speech-recognition (ASR) model [13] for live transcription. The transcripts were summarized by the T5 transformer [15], and concise multiple-choice questions (MCQs) were generated automatically and delivered immediately after the lecture through a secure Flask-based quiz interface. All models were built using Python 3.10 SDK (Software Development Kit) using the Hugging Face Transformers library and executed locally within a Docker environment on an Intel i7 workstation (32 GB RAM, RTX 3060 GPU). This setup ensured real-time operation while emulating the bandwidth and latency constraints typical of remote teaching in Table 2.

4.2 Dataset and participants

To test the dual-layer engagement validation framework, we created our own dataset that records both visual engagement signals and students’ quiz responses during live online classes. The data was gathered only for research and comparison under controlled classroom conditions. It includes matched E-scores from a commercial vision-based tool and C-scores from short quizzes given right after each lecture.

A separate set of 16,000 labelled facial expression images was also created to fine-tune a local vision model, used only for testing and comparing results with the commercial E-scores. The live classroom study itself depended entirely on the commercial engagement tool and the new comprehension layer, while this extra dataset was included to help others reproduce the work and run future comparisons. Images were captured using a OnePlus 7 Profor high-resolution input, a Realme UI 5.0 or mid-range performance, and a Lenovo 300 FHD Webcam. This variety of devices represents the mixed hardware available to students, which can influence both image quality and engagement detection accuracy.

In the comprehension layer, the same group of students took automatically generated MCQ tests right after each lecture. For every student, a Comprehension Score (C-score) was worked out by dividing the number of correct answers by the total questions, which helped show how well each learner understood the topic. The study involved students aged 10 to 32 years, covering both school and undergraduate levels, so that the framework could be tested across different age groups and learning abilities. All data were collected under the institute’s ethical approval process, and personal details were removed before analysis to maintain privacy.

The dataset was created with two clear purposes. One was to recognize engagement levels from visible behaviors, and the other was to see if those behaviors actually reflected real understanding. The combined record of E-scores and C-scores provided the key information for measuring how closely observed attention matched genuine comprehension, which forms the core of this framework’s validation.

4.3 Result and analysis

Figure 3 describes the working of online student engagement tool. During the live session, the engagement tool continuously captured visual cues while the lecture audio was processed in parallel. The Whisper model converted the audio to text, which was then summarized by T5 and turned into a ten-question, content-based MCQ quiz. Students completed the quiz right after the lecture, and the system calculated their Comprehension Score (C-score) as the percentage of correct answers, as shown in Eq. (1).

A C-score of 70 or above was taken as a sign of good understanding, which matches the usual mastery level used in education. The summaries and quizzes created by the system were reviewed by two senior teachers to make sure the content was accurate and matched the learning goals. Their agreement was strong, with reliability values of κ = 0.82 for relevance and κ = 0.79 for difficulty, showing that the generated material was of good quality.

Student engagement was divided into three groups — Low (0–49), Moderate (50–74), and High (75–100). The E-scores and C-scores were compared using Pearson’s correlation (r) to find how closely visible attention matched actual learning. The whole process ran automatically in the background, without interrupting the lecture or disturbing the students.

Figure 3. Experimental setup for validating online student engagement

Table 3. Sample results mapping: Comprehension vs engagement scores

Table 3 shows sample results for ten students chosen at random. In several cases, the level of visible engagement did not match the actual understanding. For example, student S008 had an E-score of 80 (High) but a C-score of 52 (Moderate), while student S006 showed the opposite pattern with E = 69 (Moderate) and C = 81 (High). These differences make it clear that paying attention on camera does not always mean the content was fully understood.

When all records were analysed, the Pearson correlation between E-scores and C-scores was r = 0.31, showing only a weak link between the two. As seen in Figure 4, the scatter plot spreads widely with just a slight upward trend, which means that webcam-based attention does not consistently reflect real understanding. These results highlight why a content-aware and performance-based layer is important for making sense of engagement data.

Figure 4. Correlation scatter plot – E-score vs C-score

This section compares the Engagement Scores (E-scores) from visual tracking with the Comprehension Scores (C-scores) produced by the dual-layer framework. The analysis covers basic statistics, correlation results, both multi-class and binary classifications, and cases where visible attention and actual understanding do not match. All findings are based on data from 100 students who took part in ten live lectures conducted under the setup described earlier.

6.1 Comprehension–engagement mapping

Each lecture was tracked by an automatically generated post-lecture quiz using the T5 model. Continuous E-scores (0 to 100) were obtained from commercial vision-based tools such as GoGuardian and Engageli, while C-scores were derived from the students’ performance in quiz.

For consistency with engagement software outputs, three categorical bands were adopted for both metrics i) Low (0 to 49), ii) Moderate (50 to 74), and iii) High (75 to 100). For binary evaluation, thresholds were defined as Engaged (E ≥ 75) and Actually Engaged (C ≥ 70).

Representative examples demonstrate the deviation between behavioral and cognitive indicators. Student S007 achieved a high E-score of 82 yet obtained a C-score of 49, reflecting weak comprehension despite seeming focus. Conversely, Student S004 recorded moderate engagement (E = 60) but a C-score of 91, indicating deep understanding despite limited visible cues. Such discrepancies validate the core hypothesis that visual attentiveness alone is an untrustworthy factor for cognitive engagement.

6.2 Correlation analysis

Across all lectures, the Pearson correlation between continuous E-scores and C-scores was r = 0.31 (p < 0.05), showing only a weak positive link between visual attention and actual comprehension. As seen in Figure 4, the scatter plot shows a wide spread of data points around the regression line, meaning that higher E-scores only slightly relate to higher C-scores. This pattern supports previous research suggesting that webcam-based measures mainly capture surface-level attention. In numerous cases, students who appeared highly engaged on camera performed poorly on comprehension tests, while others with moderate engagement showed strong understanding. These differences clearly highlight the gap between observed behaviour and real cognitive learning that still exists in many AI-driven monitoring systems.

6.3 Classification performance

To compare engagement and comprehension across clear categories, both E-scores and C-scores were grouped into three levels — High, Moderate, and Low. The summary of these classification results is shown in Table 5.

The findings show that the vision-based monitoring system was fairly reliable in matching engagement with comprehension results. Even so, about one-third of the cases showed mismatches, where students either appeared highly engaged but understood less, or seemed less attentive yet showed strong comprehension.

Using binary thresholds (E ≥ 75; C ≥ 70), the confusion matrix presented in Figure 5 and Table 5 quantifies this relationship. The model achieved an overall accuracy of 68.7%, with precision = 0.72, recall = 0.65, and F1 = 0.68.

Table 5. Sample results mapping: Comprehension vs engagement scores

Figure 5. Confusion matrix – engagement classification vs. comprehension assessment

6.4 Discussion and findings

The weak correlation (r = 0.31) and reasonable accuracy suggest that behavioural E-scores on their own do not give a full picture of learning. A student may appear focused on camera but still fail to grasp the topic, while another who looks less attentive—or even works off camera—might understand the lesson very well.

Adding a comprehension-based check benefits cut down these wrong classifications and turns basic behavioural data into insights that teachers can actually use. This shows that the proposed dual-layer framework makes AI-based engagement analysis more meaningful and trustworthy by connecting what is seen on camera with what students truly understand.