Multi-Modal Behaviour Monitoring in Virtual Assessments Using ProctorGuard AI

This research aims to design an anomaly detection system with multiple behavioural signals (gaze direction & lip sync) for the improvement of remote proctoring by detecting two common types of misconduct in virtual testing environments: off-screen distractions and lip sync impersonation/cheating. The development of this real-time multi-modal anomaly detection framework will be supported by a comprehensive literature review on previous work in both areas (behavioural signal processing and fraud prevention).

2.1 Gaze tracking and behavioural monitoring

Gaze estimation research spans more than two decades, and tracing its development reveals a pattern worth noting: a gradual departure from controlled laboratory conditions toward the unpredictable circumstances of ordinary use. That departure, as this section argues, remains incomplete where exam proctoring is concerned, and the various technologies used are pointed in Figure 1.

Using stereo vision to recover head orientation and gaze direction within three-dimensional space provides a sensible starting point [1, 2]. Achieving solid geometric precision came at the cost of a calibrated multi-camera rig, limiting practical adoption outside research settings. Subsequent efforts focused on eliminating these hardware dependencies. Key advances include easing calibration requirements while allowing free head movement [3], as well as addressing variable illumination [4]. Accounting for variable lighting is particularly vital in remote examinations, where uncontrolled ambient conditions and consumer webcams are standard. Evaluating these systems under such domestic conditions remains an open challenge, leaving it unclear whether these gains transfer effectively to consumer hardware or merely displace the problem.

A second, largely independent strand of research prioritised accessibility over accuracy. Embedded eye-detection systems [5, 6], contactless tracking hardware [7], and gaze-driven assistive interfaces [8] pushed sensing capability onto cheaper, smaller devices, and this body of work is what makes browser-based monitoring feasible at all. Yet accessibility of this sort tends to come at a price rarely stated outright. A simplified optical setup running on a low-resolution consumer sensor [9] may suffice for coarse judgments, such as detecting whether eyes are open, but distinguishing a glance toward scratch paper from a covert glance toward a hidden phone is a considerably harder task for the same equipment. Webcam-based trackers built on this foundation [10, 11] confirmed that real-time monitoring on ordinary hardware was achievable, but achievability of tracking says nothing about sufficiency for detecting dishonest behaviour, and the two claims are frequently treated as interchangeable when they are not.

Demonstrating that multivariate normal mixture models can run in real time over high-dimensional hyperspectral imagery via GPU parallelisation without sacrificing accuracy provides a foundational assumption for later anomaly detection systems [9]. Although developed for remote sensing rather than remote proctoring, this framework has largely been adopted by webcam-based systems without direct verification. Transitioning from spectral pixel arrays to low-resolution facial video introduces significant differences in noise profiles, dimensionality, and temporal behaviour. Validating whether this favourable trade-off between throughput and precision holds across that domain transition remains an unexamined gap within the remote examination literature.

Developing anomaly detection for gaze and conduct has followed three successive methodologies. Initial efforts relied on explicit, rule-based heuristics: deploying adaptive thresholds to trigger behavioural alerts [12] or applying geometric formulations to detect deviations in visual attention. While offering low computational overhead and high interpretability, these rule-based strategies [13] remain brittle to environmental shifts, as parameters tuned for specific camera angles, illumination levels [14, 15], or user cohorts generalise poorly without cross-condition validation. A subsequent line of work transitioned to learned representations, notably by implementing YOLOv4 to automate gaze-zone classification [16]. Although moving from heuristic rules and geometric models to deep architectures aligns with broader trends in computer vision, benchmarking these three paradigms against one another on a unified exam-monitoring dataset remains unaddressed. Lacking such a comparative evaluation, justifying the heightened computational demands, training overhead, and architectural complexity of deep learning classifiers over well-calibrated geometric baselines remains difficult.

Separately, open and wearable platforms contributed to accessibility in a different sense. The Pupil platform [13] and the iShadow wearable device [14] lowered the entry barrier for gaze research broadly, giving researchers without proprietary equipment a way to build and test their own applications. Their contribution to the field at large is not in question. Their relevance to proctoring specifically is less certain, since both were designed around a cooperative subject rather than one actively attempting to circumvent detection, precisely the scenario an exam-integrity tool must anticipate.

Integrating these separate methodological threads into end-to-end deployable pipelines represents the current state of practical exam monitoring. Developing browser-based architectures for real-time proctoring [17, 18], as well as combining landmark-based facial modeling with dynamic behavioral profiling [19, 20], reflects the practical frontier of unifying accessible hardware, live inference, and automated anomaly detection.

A primary methodological limitation across these systems is the omission of false positive rates under benign, everyday test-taking behaviours, which are frequently masked within aggregate accuracy metrics. Common, non-fraudulent actions—such as unfocusing the eyes during cognitive reflection, shifting head posture, glancing away to recall information, or checking the time—remain poorly differentiated from deliberate academic dishonesty under current detection logic [12, 15, 16, 19, 20]. Validating this boundary under naturalistic conditions remains critical, as high false alarm rates carry severe academic consequences, rendering systems with high nominal accuracy unreliable for responsible deployment.

Taken as a whole, this body of research, as mentioned in Table 1 shows that the individual pieces needed for automated exam proctoring—dependable gaze estimation, hardware accessible enough for everyday circumstances, anomaly detection capable of running at scale in real time, and increasingly capable behavioural classification—have each reached a reasonable degree of maturity on their own. What remains missing is convincing proof that these pieces work well once joined together: a system built entirely on consumer webcam equipment, resilient to the lighting and posture variation typical of a home environment, that can also be shown, under rigorous testing against genuinely innocent variation rather than staged dishonesty alone, not to penalise students for the ordinary conduct of sitting an exam. Addressing that gap in integration and testing, rather than any single missing technique, is the aim of the present study.

Figure 1. Technology applications in eye tracking

Table 1. Summary of gaze-based monitoring techniques

2.2 Lip-sync detection and audio-visual anomaly analysis

Deepfakes, fake videos that can be created using AI technology, have caused concern because they can be made to look like someone else, contain video from another time, or even be completely synthetic (man-made) speech. The reason this is such a big deal is that it can be used to create an "impersonation" on an online platform, make fake answers appear as if they were given at different times, or produce fake synthesised speech. Audio-video desynchronisation has been identified as one of the main indicators that a video may be a deepfake [21]. A thorough review provides an overview of the use of both neural networks and 3D modelling in order to synchronise a person's lips with their voice [22]. In addition, the use of a Vision Transformer has been proposed to identify small differences in the way a person's mouth moves when speaking [23].

Reviewing lip-sync generation techniques highlights a specific focus on achieving high levels of synchronisation fidelity between spoken language and video media across multiple languages and in real time [24, 25]. The use of anomaly detection via lip contour analysis has been proposed [26], alongside fusing Audio-Lip modalities with few-shot learning to enable adaptation to low-data environments [27]. Building upon these concepts, additional methods have been proposed for identifying audio-visual mismatch in deepfake defence [28].

Contributions supporting the above ideas include showing that using Segmented Facial References improves Mouth-Speech Alignment [29], as well as developing a pretrained system called SAM-Wav2Lip++, which uses behavioural realism and lip-sync integrity to evaluate video content [30]. As such, these methods can be applied directly to speech-monitoring modules in proctoring systems.

There is clearly an emerging trend toward utilising real-time gaze tracking and lip-sync verification to improve the integrity of remote exams, as demonstrated in the literature reviewed. The evidence from the research indicates that gaze-tracking technology has significant promise in helping detect inattention and/or dishonesty during remote exams, using low-cost options (e.g., webcam, free/open-source software). Lip-sync verification is a vital method to identify impersonation and/or pre-recorded answers, especially when using sophisticated architectures like Vision Transformers or audio-visual fusion networks. Combining both methods provides a stronger, more reliable system for the simultaneous, real-time monitoring of a test-taker's visual focus and the authenticity of their spoken words. Both of these additional layers will support the use of adaptive thresholds and dynamic behavioural modelling to help improve the accuracy and sensitivity of detecting anomalies. However, despite this progress, the field is still lacking in the development of standardised benchmarks to allow researchers to compare the results of different studies. Therefore, although there are many promising research directions toward developing robust, real-time, and multi-modal proctoring systems, the research continues toward developing real-time, explainable, and multi-sensory systems for ensuring fairness and trustworthiness in remote assessments. This literature review has provided strong evidence supporting the aims of the proposed research to develop an integrated system combining gaze and lip-sync-based anomaly detection for remote proctoring applications.

The surveyed literature identified some general constraints on the development of systems that utilise real-time gaze. tracking and lip-sync verification to enhance the integrity of remote assessments. Gaze tracking has been demonstrated in the literature to be an effective technology to identify inattention and/or dishonesty during remote exams, including using low-cost solutions such as a standard webcam and open-source platforms. Additionally, lip-sync validation has emerged as a critical tool for identifying impersonation and/or pre-recorded responses, particularly when supported by advanced architectures such as Vision Transformers and audio-visual fusion networks. When used in combination, these two types of data support the creation of more robust and reliable systems that enable the simultaneous, real-time monitoring of both the visual attention of test-takers and the authenticity of their spoken words. The inclusion of adaptive thresholding and dynamic behavioural modelling enhances the ability of these systems to detect anomalies more sensitively and accurately.

In spite of the advancements outlined above, the literature summarised in Table 2 highlights several general constraints on the development of these systems. One major constraint is the lack of standardised benchmarks for evaluating the effectiveness of proctoring systems, which complicates comparing results among studies. Most proctoring systems are tested under controlled/semi-controlled conditions, which do not always represent the full range of possible conditions found in real-world online assessments. Variability in hardware, lighting conditions and diversity in user behaviour may also impact detection accuracy. Additionally, long-standing issues of privacy and explainable AI continue to challenge the development of proctoring systems. Collectively, these constraints highlight the need for continued research into the development of robust, real-time, and generalisable multi-modal proctoring system.

Table 2. Summary of lip-sync verification and audio-visual anomaly detection

Proctor Guard AI report provides a comprehensive overview of the testing outcomes for both the lip-sync detection and eye gaze tracking functionalities, demonstrating the system's ability to monitor and detect potential anomalies during virtual assessments.

4.1 Lip-sync detection evaluation

The lip-sync detection module’s performance was evaluated using four controlled testing scenarios: Audio with lip movement absent; Humming with lip movement; Lip movement absent with Audio; Synchronized video (standard) for comparison. The four test cases are representative of the common anomalies experienced by online proctoring systems.

4.1.1 Audio w/o lip movement

Audio was continuously played for each subject, yet no lip motion occurred. Speech detection via VAD was constantly positive (True) while Mahalanobis distance based on lip motion was always negative (False) in terms of synchronization. All frames were identified as such (Table 1), and demonstrated the ability to detect pre-recorded/externally played audio as an example of cheating activity. Accuracy for this test case was 100% and validated complete anomaly detection for this specific test case.

4.1.2 Humming audio & lip movement

Candidates were instructed to continuously hum while exhibiting lip movement. It is expected that VAD would fail to identify humming as speech, and thus, the Mahalanobis distance would remain relatively low because of the presence of slight lip movement. In addition to those frames which indicated False on both VAD and Mahalanobis distance columns (true negatives), there were also frames where one or both columns indicated True (Table 2). These results indicate that humming represents less risk than spoken impersonation; however, it will still be identified under partial mismatch.

4.1.3 No audio & lip movement

Lip movement occurred, but candidates had no audio input. The VAD column provided correct results by identifying all frames as False, while the Mahalanobis distance indicated True to confirm significant lip movement relative to normal voice production (Table 4). Candidates who mute their microphones to prevent monitoring may exhibit such behaviours as they speak. Once again, the system identified these inconsistencies properly to validate the detection logic of the model.

4.1.4 Standard video (normal)

Standard, synchronised speech and lip movement were used to evaluate the baseline performance of the system. Both the VAD and Mahalanobis metrics provided positive results (True) to confirm that the behaviour exhibited was normal (Table 4). The use of standard video for comparison purposes helped to further validate the low false-positive rate of the model.

Table 4. Performance comparison of lip synchronisation and audio detection across different video scenarios and system windows (lipsync accuracy)

4.2 Lip-sync accuracy analysis

A general overview of detection performance across multiple test video cases using a sliding window method to compare video cases, all with the same 0.3-second speaking delay window, indicated 100% detection accuracy for those test cases which are clearly indicative of deceptive behaviour; i.e.:

• No Audio + Lip Sync
• Audio + No Lip Movement

The system has a high rate of reliable flagging for these test cases. Conversely, test cases representing normal video samples (i.e., YouTube videos with real speakers) produced extremely low mismatch percentage results (some of which were as low as 0%), indicating virtually no false positive results from real test video cases.

Also, although the system had lower accuracy for borderline cases such as partial lip sync or delayed audio, the system's ability to detect deception remained at an accuracy rate of above 85%, regardless of the background noise or other conditions present within the test cases.

4.3 Gaze estimation results

ProctorGuard AI's Gaze Tracking Component was tested for its ability to measure a user's focus on the screen and identify when a user deviated from viewing the screen during Virtual Assessments. This component worked well in all of the test scenarios. Specifically, it correctly recognised when a user made a left or right gaze shift while looking away from the edge of the screen, and when the user made an upward gaze (it utilised specific threshold values to recognise keyboard activity when applicable). More importantly, the system used a built-in function to map the user's gaze vector to the user-defined screen boundaries to flag any time the user gazed outside of the screen boundaries ("OUT OF SCREEN").

The Calibration Phase of the Gaze Tracking Module, which involved the user focusing their eyes on the four screen corners, allowed for defining the area of the screen that would be monitored by the system and greatly enhanced the system's ability to accurately map the user's gaze to the screen. Figure 5 is a visual example of how the system tracked the user's gaze in real-time during live testing (identified by "Calibration" and "Result"). The data collected demonstrates that the Gaze Tracking Module was able to clearly distinguish between typical user behaviour and other forms of distraction or non-compliance. When combined with Lip-Sync Detection, this module will enhance the overall monitoring capabilities of the system, resulting in a robust and low false positive rate, real-time proctoring solution.

Figure 5. Sample output for gaze estimation results

4.4 Discussion and inference

A composite assessment of the lip-sync detection and Gaze Tracking Modules is an indicator that the ProctorGuard AI system is a viable means to enhance the validity of virtual exams by providing a valid measurement of behaviour during proctored testing. Lip sync detection had very high accuracy in determining discrepancies between voiced activity and lip movement, particularly in critical scenarios (i.e., silent lip movement and muted speech) in which detection was nearly perfect. Gaze tracking also provided reliable identification of deviations from normal viewing patterns (i.e., left, right, up, off-screen) and distinguished between keyboard activity by using vertical gaze thresholds. The two modules combine to form a solid behavioural monitoring framework. Real-time, low-latency detection of both visual and auditory anomalies was possible due to vector-based calibration of gaze and the use of Mahalanobis distance on lip motion data for detecting cheaters or individuals who lose focus, while minimising false positive responses. The results confirm that ProctorGuard AI can detect cheating attempts or loss of focus in remote interview environments and online exam environments through subtle visual and auditory cues; therefore, it represents a comprehensive and scalable solution for securing those environments.

Written informed consent was obtained from the author shown in the figure for participation in the gaze estimation experiment and for publication of the identifiable facial images in this article.

This project has been funded and assisted by the Centre for Artificial Intelligence at Christ (Deemed to be University) through its institutional support, mentoring, and facilities. In addition, we are grateful for the input from the AI Guild student group and for the collaborative and supportive efforts of the School of Sciences in relation to this endeavour.