EfficientNet-KNN for Real-Time Driver Drowsiness Detection via Sequential Image Processing

Driver drowsiness represents a significant risk factor in road safety, causing momentary attention lapses that can lead to severe accidents [1, 2]. Recent studies indicate that drowsiness-related incidents account for a substantial proportion of traffic accidents globally, with significant economic and human costs [3]. This phenomenon is particularly prevalent during prolonged driving in monotonous traffic conditions, where drivers often underestimate their fatigue levels and continue operating vehicles despite increasing impairment [4]. As highlighted by Kumar and Tarei [4], unsafe driving behaviors significantly contribute to road accidents, with driver drowsiness being a key contributing factor.

Technological interventions, particularly those leveraging artificial intelligence and computer vision, present promising solutions to this critical safety challenge [5, 6]. The application of deep learning techniques for automated drowsiness detection represents a proactive approach to enhancing road safety through early warning systems [7]. Recent advances in this field have demonstrated remarkable potential, with CNN-based approaches achieving accuracy rates exceeding 95% in controlled environments [8]. These AI-powered systems analyze data from in-vehicle cameras to detect preliminary signs of drowsiness, enabling timely alerts and preventive measures. As demonstrated by Gwak et al. [6], ensemble machine learning approaches using hybrid sensing have shown particular promise in early drowsiness detection.

Current state-of-the-art approaches in drowsiness detection demonstrate varying levels of success. Deep learning-based methods, such as Magán et al.'s [9] ADAS system, show the potential of analyzing temporal image sequences, though achieving only moderate accuracy (65%). More advanced CNN implementations, as shown by JR et al. [8], have achieved 95.3% accuracy in controlled environments. Physiological signal-based approaches, exemplified by Ayatollahi et al.'s [10] ECG analysis, offer alternative detection mechanisms but face practical implementation challenges in vehicular environments. While these studies advance the field, they often prioritize accuracy over computational efficiency, limiting their practical deployment in real-time automotive systems.

Despite these advances, current drowsiness detection systems face several critical limitations. First, many approaches analyze individual frames in isolation, failing to capture the progressive nature of fatigue manifestation in facial expressions over time, as noted in recent temporal analysis studies [11, 12]. Second, existing solutions often require substantial computational resources, making real-time deployment challenging in resource-constrained automotive environments [13]. Third, the trade-off between detection accuracy and processing speed remains inadequately addressed, particularly for sequential image analysis. These limitations are further complicated by variations in environmental conditions and individual differences in drowsiness expression patterns [14], highlighting the need for more robust and efficient detection approaches.

This paper addresses these challenges through three primary contributions: 1) Development of a novel architectural integration combining EfficientNet's feature extraction capabilities with KNN classification for optimal drowsiness detection, building upon recent advances in efficient deep learning architectures [15, 16]; 2) Implementation of an efficient sequence processing approach that captures temporal patterns in facial expressions while maintaining minimal computational overhead, addressing the limitations identified in previous temporal analysis studies [17, 18]; 3) Introduction of a comprehensive evaluation framework demonstrating significant improvements in both accuracy (99.52%) and computational efficiency (49.9× faster than baseline approaches).

Our approach uniquely leverages EfficientNet's compound scaling properties for feature extraction while utilizing KNN's effectiveness in classification, particularly for sequential data patterns indicative of drowsiness progression.

The remainder of this paper is organized as follows: Section 2 provides a comprehensive review of drowsiness detection approaches, highlighting the evolution of methodologies in this field. Section 3 details our technical approach, including the integration of EfficientNet feature extraction with KNN classification and our sequence processing methodology. Section 4 presents experimental results demonstrating substantial improvements in both accuracy and computational efficiency. Section 5 discusses the implications of the findings for automotive safety systems, while Section 6 concludes the study and highlights avenues for future research.

The primary objective of our research methodology was to develop an algorithm capable of achieving optimal performance in drowsiness detection while maintaining a compact model size and rapid response time. We focus on analyzing the driver's facial features as the central factor in detection, with an emphasis on efficiency to deliver practical solutions for real-world implementation.

Figure 1 illustrates our research framework, encompassing data acquisition and pre-processing, data separation, model design using EfficientNet and KNN approaches in sequence processing, validation using k-fold cross-validation, and comprehensive model evaluation.

101.png

Figure 1. Research framework for drowsiness detection

3.1 Dataset and data acquisition

This research utilizes the ULg Multimodality Drowsiness Database (DROZY database) as the primary data source [26]. This multimodal dataset comprises EEG signals, facial expression videos, and subjective assessments using the Karolinska Sleepiness Scale (KSS). The dataset contains 36 videos: 14 depicting non-drowsy conditions and 22 showing drowsy states. Each video is approximately 10 minutes in duration, with a frame resolution of 512×424 pixels, stored in mp4 format at frame rates ranging from 15 to 30 frames per second (fps). These videos were carefully selected to provide diversity and accurately represent drivers' drowsy states, enabling effective detection of various fatigue-related facial expressions.

3.2 Data preprocessing

In the initial preprocessing stage, each video is segmented into 5-second intervals using a windowing approach [27]. This timeframe was selected because microsleep episodes—brief involuntary sleep occurrences—typically manifest within this duration. Moreover, the 5-second temporal window was selected based on neurophysiological evidence and automotive safety requirements, rather than arbitrary choice. 1) This selection is supported by multiple converging factors: Clinical studies demonstrate that microsleep events typically occur within 1-15 seconds, with peak frequency around 3-8 seconds [28]; 2) Drowsiness-related facial changes (progressive eyelid closure, reduced blink frequency, head nodding) manifest over 3-7 second intervals before reaching critical fatigue states [29]; 3) Driver response capability significantly deteriorates within 4-6 seconds preceding sleep onset, making this window critical for intervention timing [30].

Thus, For each 5-second segment, we capture 25 consecutive image frames to document the driver's facial expressions throughout the period, providing diverse facial cues for model training.

Subsequently, each captured image is resized to 128×128 pixels [31, 32] to reduce computational complexity and improve processing efficiency while preserving essential visual information. The Haar Cascade method is then applied to detect faces in each image [33], ensuring the model focuses specifically on facial features relevant to drowsiness detection.

The Haar Cascade technique, based on Haar features, detects specific objects by analyzing light and shadow patterns in images. The method was selected for its effectiveness in real-time object detection [34]. Haar Cascade employs Haar filters, integral images, and cascade phases during the training process. The Haar filter can be visualized as a rectangular function with positive values in one half and negative values in the other, expressed as:

$H(x, y)=\left\{\begin{aligned} 1, & { if\ the\ pixel }(x, y) {is\ inside\ the\ eye\ are } \\ -1, & { others }\end{aligned}\right.$               (1)

where, x and y represent pixel coordinates in the two-dimensional image space. In drowsiness detection, these coordinates help identify pixel locations in the eye area—a critical region for detecting fatigue indicators such as slow eye movements or closure.

The integral image (II) calculation facilitates rapid determination of pixel counts in specific image regions using the cumulative formula:

$I I(x, y)=\sum_{i=0}^x \sum_{j=0}^y I(i, j)$              (2)

where, I(i,j) represents pixel intensity at position (i,j), and II(x,y) is the cumulative result up to coordinates (x,y). This formula provides an efficient representation for calculating pixel quantities, accelerating the Haar Cascade detection process [35].

Figure 2 shows visual representations of facial frames in the driver drowsiness detection process, while Figure 3 illustrates the results of applying the Haar Cascade method to the image data.

Following face detection, the processed image data is normalized by dividing pixel values by 255 [36], adjusting the pixel range from 0-255 to 0-1. This normalization standardizes the data to align with model training requirements, particularly for activation functions that operate optimally with inputs in specific ranges [37, 38].

102.png

Figure 2. Visual representations of facial frames in the driver drowsiness detection process (a) normal; (b) drowsiness

103.png

Figure 3. (a) Pre-processing visualizations and (b) Haar cascade method results on image data

The prepared dataset is then divided into training (70%) and testing (30%) sets [39], with label stratification to maintain consistent class distribution across subsets. Additional experiments were conducted with varying dataset proportions (100%, 75%, 50%, and 25% of the initial dataset) to investigate the impact of dataset size on model performance [40].

3.3 Model deployment

Our study aims to enhance eye state classification accuracy (open versus closed), exceeding traditional detection systems. The research progresses through two model development strategies: first, implementing a K-Nearest Neighbors (KNN) model; second, employing EfficientNet for feature extraction followed by comprehensive model comparison.

3.3.1 Strategy I: Model construction with the K-Nearest Neighbors algorithm

The first strategy employs the KNN algorithm for drowsiness detection. KNN is a machine learning classification method that considers the majority class from a set of K-Nearest Neighbors [41, 42]. In our context, KNN identifies facial expression patterns associated with drowsiness by comparing similarities with previously acquired training data [43]. Figure 4 presents the complete architecture for this approach.

104.png

Figure 4. Classification process using Strategy 1: K-Nearest Neighbors (KNN)

KNN is expected to achieve high accuracy in recognizing specific drowsiness indicators such as closed eyes or slower facial movements. The algorithm classifies items based on their similarity to training data using the Euclidean distance formula:

$d(A, B)=\sqrt{\sum_{i=1}^n\left(A_i-B_i\right)^2}$                 (3)

where, Aᵢ and Bᵢ are the i-th components of vectors A and B, and n is the number of features. For classification, KNN identifies the K closest training instances and assigns the class based on majority voting:

$y^{\prime}=\operatorname{argmax}_c\left(\sum_{i=1}^K \delta\left(c_i, c\right)\right)$             (4)

where, argmax_c is the class with the largest count among K nearest neighbors, cᵢ is the class of the i-th neighbor, and δ is the Kronecker delta function (1 if cᵢ = c, 0 otherwise).

For high-dimensional facial feature data, small K-values are preferred to preserve local pattern sensitivity crucial for subtle drowsiness indicators, ability to reduce computational overhead for real-time processing requirements, also maintain discriminative power in sparse high-dimensional spaces [44].

3.3.2 Strategy II: Model construction with KNN and feature extraction using EfficientNet

Our second strategy integrates EfficientNetB0 for feature extraction with KNN for classification. EfficientNetB0 is utilized to extract meaningful features from facial image frames [15, 16].

EfficientNet as feature extraction. EfficientNet represents a family of convolutional neural network models that achieve state-of-the-art accuracy with significantly fewer parameters than traditional architectures. The model employs compound scaling to systematically balance network depth, width, and resolution dimensions. EfficientNetB0, the baseline architecture in this family, utilizes mobile inverted bottleneck convolution (MBConv) as its core building block, incorporating squeeze-and-excitation optimization.

In our implementation, EfficientNetB0 serves as a feature extractor that processes input images and produces rich feature representations. The model's efficient design allows it to capture complex patterns in facial features while maintaining computational efficiency—a crucial consideration for real-time drowsiness detection systems.

Time distributed layer. Since driver videos contain sequences of frames demonstrating temporal changes in facial expressions, we implement a "time-distributed" layer in the EfficientNetB0 architecture [17]. This specialized layer enables sequential feature extraction on each facial frame in the video, facilitating temporal understanding of facial expression evolution [18].

The time-distributed layer wraps the EfficientNetB0 base model, applying it to each temporal slice of input separately. This approach preserves the temporal relationship between consecutive frames while extracting spatial features from each frame. By processing the input as a sequence rather than independent frames, the model can detect temporal patterns indicative of drowsiness development, such as progressive eye closure or head position changes across multiple frames [45].

Global Average Pooling. Following feature extraction, Global Average Pooling (GAP) is applied to reduce the dimensionality of the extracted features [46]. GAP calculates the average feature values across the entire spatial domain of the image, producing a more condensed representation while preserving essential information [47].

In our EfficientNetB0 implementation, GAP is applied after feature extraction to reduce data complexity while retaining discriminative information relevant to drowsiness detection. This condensed feature representation serves as input for the subsequent KNN classification, effectively combining EfficientNetB0's feature extraction capabilities with KNN's classification strengths [48].

Figure 5 illustrates the complete architecture for this integrated approach.

105.png

Figure 5. Classification process using Strategy 2: K-Nearest Neighbors (KNN) with feature extraction using

106.png

Figure 6. feature extraction visualization using 3-component PCA for awake and drowsy

Feature extraction results with 3-component PCA. To visualize the effectiveness of our feature extraction process, we employed Principal Component Analysis (PCA) to reduce the high-dimensional feature space to three principal components. Figures 6 and Table 1 present the visualization of these components for both awake and drowsy states.

Figure 6 demonstrates the distribution of extracted features in 3D space after dimensionality reduction through PCA. The clear separation between awake (green) and drowsy (red) states validates the discriminative power of our feature extraction approach. This visualization confirms that the features extracted by EfficientNetB0 contain sufficient information to distinguish between the two classes.

Table 1 presents detailed statistical distributions of the extracted features for both classes (y=0 for awake, y=1 for drowsy). The statistical properties of each principal component demonstrate distinct characteristics between the two classes. For the first principal component (PC1), awake subjects show a mean of 2.63 compared to -1.69 for drowsy subjects. PC2 exhibits an inverse relationship with means of -1.03 for awake versus 0.66 for drowsy states. PC3 follows a similar pattern with means of -0.28 for awake versus 0.18 for drowsy states. The variability (standard deviation) is comparable between classes across all components, with slightly higher variance in the awake state. These statistical differences support the discriminative capability of our feature extraction methodology, providing quantitative evidence that the extracted features contain sufficient information to differentiate between drowsiness states.

Table 1. Statistical distribution

Following model development for each strategy, we employed K-Fold cross-validation with stratification (K=3) to evaluate model performance. This approach divides the dataset into three alternating subsets while maintaining balanced label distribution in each fold, providing robust assessment of model generalizability.

3.4 Model evaluation

We employed multiple performance metrics to evaluate our drowsiness detection models. Accuracy measures the algorithm's overall correctness in classifying the entire dataset [49]. Precision indicates the model's ability to generate accurate positive predictions of drowsiness while avoiding false positives [50]. Recall (sensitivity) assesses the model's capacity to identify all genuine instances of drowsiness. The F1 Score balances precision and recall, offering comprehensive insight into model performance [50].

Additionally, execution time metrics are critical for drowsiness detection systems intended for real-time applications. The model's ability to generate predictions promptly ensures timely response to driver fatigue, potentially preventing accidents.

Our experimental results demonstrate that combining EfficientNet for feature extraction with KNN for classification (EFF+KNN) produces superior performance in driver drowsiness detection compared to using KNN alone. Several key factors contribute to this enhanced performance:

5.1 Model architecture and feature representation

The significant improvement in both accuracy and execution time for the EFF+KNN model can be attributed to EfficientNet's ability to extract highly discriminative features while reducing dimensionality. The original feature space of 1,228,800 dimensions (25×128×128×3) was reduced to 1,280 dimensions through EfficientNet's architecture, resulting in a more compact yet informative representation.

This dimensionality reduction directly impacts execution time, with the EFF+KNN model processing data approximately 50 times faster than the KNN-only approach. Such efficiency is crucial for real-time drowsiness detection systems where prompt alerts can prevent accidents. The statistical significance of these performance differences, confirmed through t-tests (p < 0.05), validates the superiority of the combined approach.

5.2 Temporal context benefits

The integration of time-distributed layers in our EFF+KNN model enables sequential processing of facial frames, capturing temporal relationships that are essential for drowsiness detection. This temporal awareness allows the model to recognize progressive patterns such as gradual eye closure, head nodding, or delayed blink recovery—subtle indicators of increasing fatigue that might be missed in frame-by-frame analysis.

The confusion matrix results further support this benefit, with high precision and recall for both classes (Awake and Drowsy). The model's ability to maintain high accuracy while significantly reducing false positives and false negatives demonstrates its robustness in real-world conditions.

5.3 Dataset size implications

Our experiments across varying dataset proportions (25% to 100%) reveal interesting insights about model scalability. Both models showed performance improvements with increased data availability, but the EFF+KNN model demonstrated stronger performance even with limited data. This suggests that the feature extraction capabilities of EfficientNet effectively capture essential drowsiness indicators even from smaller datasets, making it particularly valuable for applications where data collection might be constrained.

5.4 Practical applications

The high accuracy (99.52%) and rapid execution time (0.55 seconds) of our EFF+KNN model have significant implications for practical deployment in vehicles. The model's compact size and computational efficiency make it suitable for integration with limited-resource hardware in automotive systems, potentially enabling widespread adoption of drowsiness detection technology.

The minimal false positive rate is particularly important, as frequent false alarms could lead to driver complacency or system disregard. Conversely, the high detection rate for genuine drowsiness ensures timely interventions when needed, potentially preventing fatigue-related accidents.

5.5 Limitations and considerations

Despite the impressive performance achieved, several critical limitations warrant consideration for transparent reporting and future research guidance. The current study is primarily constrained by demographic bias inherent in the ULg DROZY database, which predominantly comprises Caucasian European participants with limited age diversity (mean age ~23 years), potentially affecting model generalizability across diverse ethnic populations and age groups. This demographic homogeneity may limit detection accuracy for individuals with different facial structures, eyelid morphologies, and cultural variations in fatigue expression patterns that are common in global automotive applications. Additionally, the controlled laboratory evaluation environment may not fully represent real-world driving conditions, including dynamic lighting variations, environmental factors such as vehicle vibrations and weather conditions, and the absence of actual driving stress that could influence fatigue manifestation patterns. The model's reliance on facial features makes it vulnerable to common occlusions such as sunglasses, face masks, or hair coverage, while the single-modal vision-only approach excludes potentially valuable drowsiness indicators from steering patterns, physiological signals, or voice analysis.

To address these limitations, several mitigation strategies are proposed: (1) Demographic diversity enhancement through collaboration with international research institutions to develop multi-ethnic datasets spanning diverse age ranges and professional driver populations, coupled with transfer learning approaches for demographic-specific model adaptation; (2) Environmental robustness improvement via comprehensive data augmentation strategies including synthetic lighting condition generation, weather effect simulation, and multi-environmental field validation across different geographic locations and vehicle configurations; (3) Technical enhancement through multimodal integration incorporating steering pattern analysis, physiological monitoring, and adaptive temporal window sizing for individual-specific fatigue progression modeling; and (4) Validation robustness enhancement using independent cross-cultural datasets and longitudinal studies to assess seasonal variations and individual adaptation patterns. While these limitations must be considered when interpreting results, the study provides valuable contributions in demonstrating the first systematic evaluation of EfficientNet-KNN integration for drowsiness detection, establishing a statistical framework for method comparison, and achieving computational efficiency suitable for real-time automotive applications that inform future development of more robust, inclusive, and practical drowsiness detection systems.