A Hybrid Deep Feature Fusion and Stacked Classifier Network for Monitoring Driver Drowsiness and Distraction

Among the most significant contributing factors to road traffic accidents is driver drowsiness and distraction, both of which substantially impair driving performance and heighten the risk of collision, thereby threatening the safety of not only the fatigued or inattentive driver but also other vulnerable road users, including co-passengers and pedestrians. Such incidents can lead to severe consequences, placing countless lives at significant risk. While official statistics often highlight fatality rates, the overall impact of drowsy driving is likely underestimated and may extend well beyond the reported figures. “Drowsy driving,” also known as fatigued or tired driving, occurs when a driver operates a vehicle while exhausted or sleepy. As people rely on cars for daily transportation, the risk of serious or fatal collisions with other vehicles or stationary objects significantly increases [1]. Venkateswarlu and Ch [2] highlighted that this condition greatly endangers drivers by raising the chances of drowsiness and the potential for accidents. Distracted driving refers to any instance in which a driver's attentional focus is diverted from the primary driving task toward secondary activities, consequently impairing their judgment, responsiveness, and vehicular control. It is recognized as a significant factor in fatal accidents and injuries on the road [3]. Distracted driving encompasses any secondary activity that interferes with the driver's primary task of safe vehicle operation, mobile device communication, food and beverage consumption, passenger engagement, adjusting in-vehicle controls and navigation systems, and abnormal head-pose orientations. Since many forms of distracted driving can be observed using cameras installed in vehicles, analyzing video frames captured during real-world driving scenarios is a logical approach for detecting such behaviors [4]. Road accidents lead to damage to property and, in some cases, fatalities. There has been a significant rise in accidents caused by distracted driving, which is a growing concern. Distracted driving is generally classified into three categories: visual (looking for items inside the car, checking a GPS, or reading a text message), manual (e.g., eating or drinking while taking hands off the wheel, adjusting the radio, or texting), and cognitive (e.g., losing focus or envisioning, having a conversation with a passenger, or thinking about personal issues). These distractions can result in significant lane deviations, reduced vehicular control, slower reactions to hazards, and a diminished awareness of the roadway surroundings compared to attentive driving [5]. Data published by the National Highway Traffic Safety Administration (NHTSA) reveal that distracted driving contributed to 3,275 fatalities in 2023, highlighting the serious risks posed by inattentive behavior behind the wheel [6]. Drowsy driving is a factor in almost 6,000 fatal accidents each year [7]. A 2024 study by the Insurance Institute for Highway Safety (IIHS) found that users of partial automation were more prone to distractions, such as phone use and eating, than manual drivers. In addition, drowsy-driving-related crashes caused 633 fatalities in the same year. These statistics underscore the pressing need for heightened public awareness and preventive measures to combat risky driving behaviors. Drowsy driving is a form of impaired driving, contributing to approximately one in five fatal motor vehicle crashes on U.S. roads [8]. Detection methodologies are broadly categorized into 3 primary modalities, encompassing behavioral observations, physiological measurements, and vehicle-derived operational data, which are collectively employed to identify distracted or drowsy driving patterns [9]. Venkateswarlu and Chirra [10] introduced a hybrid Convolutional Neural Network (CNN)-ViT architecture that synergistically combines locally extracted facial features and globally oriented attention mechanisms, achieving highly accurate classification of eye and mouth states for real-time drowsiness detection.

Despite significant progress made in deep learning-based driver monitoring systems, several critical research gaps remain unresolved:

• Limited feature representation: Single CNN models often fail to capture subtle and complex driver behaviors like small head pose changes or micro-expressions.
• Lack of effective feature fusion: Most approaches do not combine complementary spatial, semantic, and contextual features from multiple models, limiting detection accuracy.

• Poor generalization: Models trained on small or imbalanced datasets struggle to handle real-world differences in lighting, facial expressions, and head poses due to limited data augmentation.

To overcome these gaps, we propose Hybrid Deep Feature Fusion and Stacked Classifier Network (HDFSNet), a hybrid deep feature fusion and stacked classifier network that integrates multiple CNNs with ensemble learning, enhancing robustness, accuracy, and generalization for driver drowsiness and distraction detection.

Subtle driver behaviors such as micro-expressions, marginal eye closure, and small head pose deviations are represented at different spatial and semantic levels within facial imagery. Lower-level convolutional layers predominantly capture fine-grained texture and edge-based cues, including eyelid contours, lip boundaries, and localized facial muscle movements, while deeper layers encode higher-level semantic and geometric representations such as global facial structure, head orientation, and contextual pose information. Consequently, single-backbone CNN architectures tend to provide only partial feature representations, which leads to increased inter-class confusion between visually similar driver states, particularly for classes such as MiddleCenter versus MiddleLeft and yawn versus no_yawn.

To empirically validate this limitation, a comparative analysis was conducted between individual CNN-based pipelines and the proposed HDFSNet framework. Models based on single feature extractors (VGG16, ResNet50, and InceptionV3 combined with Support Vector Machine (SVM), Stochastic Gradient Descent (SGD), and Multilayer Perceptron (MLP) classifiers) achieved classification accuracies in the range of 96.89%–98.25%. In contrast, the proposed multi-CNN feature fusion and stacked classification architecture improved overall accuracy to 99.08%, while significantly reducing misclassification rates among subtle head pose and yawning classes.

Furthermore, the classifier stacking strategy functions as a form of decision-level regularization. The heterogeneous base learner model complementary and partially independent decision boundaries within the high-dimensional fused feature space, and the logistic regression meta-classifier learns an optimal combination of their probability outputs. This integration enhances robustness and generalization under variations in illumination, partial occlusion, and driver posture, thereby directly addressing the limitations of limited feature representation, ineffective fusion, and poor generalization identified in the problem definition.

HDFSNet is novel because it combines features from three different CNN models (VGG16, ResNet50, and InceptionV3) and uses multiple classifiers (SVM, SGD, MLP) stacked together with a meta-classifier for final decisions. This approach captures more varied information and improves accuracy compared to methods using a single model or classifier. It also uses automated facial region detection and data augmentation to enhance generalization. Overall, HDFSNet offers a robust, accurate, and real-time solution for detecting driver drowsiness and distraction.

• Introducing an innovative architecture that combines CNN-driven hierarchical feature extraction with ML classifiers to identify driver drowsiness and distraction.
• Data augmentation strategies were employed to mitigate class imbalance, thereby improving model robustness and generalization.
• Two datasets, encompassing four classes each for drowsiness and distraction, were integrated to enhance the classifier’s performance and ability to generalize.
• For each feature extraction architecture — VGG16, ResNet50, and InceptionV3—separate classifiers, including SVM, SGD, and MLP, were individually applied to perform the classification task.
• A hybrid model, HDFSNet, was developed by integrating VGG16, ResNet50, and InceptionV3 for feature extraction with stacked SVM, SGD, and MLP classifiers, and a meta-classifier, logistic regression, is used for final prediction.
• To evaluate the proposed method, standard classification metrics encompassing precision, recall, F1-score, accuracy, confusion matrix, Receiver Operating Characteristic (ROC), and Precision-Recall (PR) curves were employed.

The literature on drowsiness and distraction detection is reviewed in Section 2. Section 3 introduces the proposed approach and provides a detailed description of the system’s components. In Section 4, the performance of each CNN architecture is evaluated and the experimental results are discussed. Finally, Section 5 concludes the manuscript by summarizing the key findings and outlining directions for future research.

3.1 Transfer learning based model for feature extraction

The system combines three pre-trained CNNs (VGG16, ResNet50, and InceptionV3) with three classifiers (SVM, SGD, MLP) to leverage their complementary strengths in feature extraction and decision-making. Each CNN extracts feature vectors from driver images, which are then classified using the chosen algorithm, resulting in nine hybrid configurations: VGG16-SVM, VGG16-SGD, VGG16-MLP, ResNet50-SVM, ResNet50-SGD, ResNet50-MLP, InceptionV3-SVM, InceptionV3-SGD, and InceptionV3-MLP.

3.1.1 VGG16 for feature extraction

The VGG16 [17] model is a simple and effective CNN known for strong image classification performance. It uses stacked 3 × 3 convolutional layers and 2 × 2 max-pooling to capture fine image details. In this work, VGG16 is used without its top layers and applies Global Average Pooling (GAP) to produce a compact feature vector, which is then classified by SVM, SGD, and MLP. This approach provides robust features for accurately detecting driver states from facial cues.

3.1.2 ResNet50 for feature extraction

ResNet50 [18], comprising 50 convolutional layers, introduced the residual connections to improve training and capture complex features. In this system, ResNet50’s top layer is removed and GAP is used to create a fixed-size feature vector, which helps accurately distinguish subtle driver behaviors. These features are then classified by base models, enhancing overall robustness and generalization.

3.1.3 InceptionV3 for feature extraction

InceptionV3 [19] is a CNN designed for high accuracy and efficiency, using parallel filters of different sizes for multiscale feature extraction. Here, the model’s top layers are removed, and GAP is applied to produce a feature vector. This approach helps identify complex driver behaviors, with extracted features used for final classification.

3.2 Base classifiers

3.2.1 Support Vector Machine

SVM is a supervised classification technique that partitions data into classes by constructing a suitable hyperplane within a high-dimensional feature space. In this work, deep features from CNNs are input to the SVM, which uses a polynomial kernel for nonlinear separation, improving class distinction. The decision function of SVM can be defined as shown in Eq. (1).

$\mathrm{f}\left(\mathrm{x}_{\mathrm{j}}\right)=\sum_{\mathrm{m}=1}^{\mathrm{N}} \alpha_{\mathrm{m}} \mathrm{y}_{\mathrm{m}} \mathrm{K}\left(\mathrm{x}_{\mathrm{m}}, \mathrm{x}_{\mathrm{j}}\right)+\mathrm{b}$                 (1)

Here, the input features are evaluated against the support vectors x_m—which represent the critical deep features—using the kernel function K. The final weighted result includes the influence of the Lagrange multipliers α_m, the training labels y_m, and the bias b.

The polynomial kernel function utilized in this implementation is as follows in Eq. (2).

$K\left(x_m, x_j\right)=\left(\gamma \cdot x_m^\tau x_j+r\right)^d$               (2)

where, $\gamma$ is the scaling factor, r is the independent term, and d represents the polynomial degree of the kernel.

3.2.2 Stochastic Gradient Descent

SGD is an efficient streamlining method that updates model parameters using single or small batches of data, reducing memory use and speeding up training. Here, an SGD classifier with logistic regression loss was applied to deep features from VGG16, ResNet50, and InceptionV3 to quickly and effectively classify driver states. The parameter modification rule for SGD is defined in Eq. (3).

$\theta_{s+1}=\theta_s-\eta \cdot \Delta_\theta L\left(\theta_s ; x_j, y_j\right)$             (3)

The update rule for the model parameters is defined as follows: $\theta_{\mathrm{s}}$ denotes model parameters at iteration $\mathrm{s}, \eta$, learning rate, which governs the magnitude of parameter adjustments during optimization, $\mathrm{L}\left(\theta_{\mathrm{s}}; \mathrm{x}_{\mathrm{j}}, \mathrm{y}_{\mathrm{j}}\right)$, loss function evaluated on j -th training sample $\left(\mathrm{x}_{\mathrm{j}}, \mathrm{y}_{\mathrm{j}}\right), \nabla_\theta \mathrm{L}$ denotes the gradient of the loss about the parameters.

3.2.3 Multilayer Perceptron

The MLP, a foundational feed-forward neural network architecture, is characterized by one or more intermediate layers positioned between the input features and output predictions. The potential to model complex associations within the data is introduced through the use of non-linear activation functions, and each layer is entirely connected to the next. The MLP processes input features through multiple layers of transformation, as shown mathematically for two hidden layers in Eq. (4).

$\hat{y}=f\left(W_3 \cdot \sigma_2\left(W_2 \cdot \sigma_1\left(W_1 \cdot x+b_1\right)+b_2\right)+b_3\right)$                  (4)

where,

• $x$  is an input feature vector (here, deep features extracted from CNNs).
• $\mathrm{W}_1$ denotes weight matrices for the first hidden layer, $\mathrm{W}_2$ is for the second hidden layer, and $\mathrm{W}_3$ for the output layer, respectively.
• $b_1, b_2, b_3$ represent bias vectors for corresponding layers.
• $\sigma_1, \sigma_2$ are activation functions like ReLU or tanh.
• $\mathrm{f}(\cdot)$ represents the final activation function for output (e.g., softmax for classification).
• $\hat{\mathrm{y}}$ is the predicted output vector.

3.3 Hyperparameter configuration for base classifiers

3.3.1 Support Vector Machine

A polynomial kernel function defined in Eq. (2) where the scaling factor is set to γ = 0.01, the independent term is r = 1, and the polynomial degree is d = 3. The regularization parameter is fixed at C = 1.0.

3.3.2 Stochastic Gradient Descent

The SGD classifier is trained using a logistic regression loss function with an altered learning rate of η = 0.001. L2 regularization is applied with α = 0.0001, and the iteration limit is set to a maximum of 1000. Training is performed using mini-batches of size 64 to balance convergence stability and computational efficiency.

3.3.3 Multilayer Perceptron

The MLP architecture consists of two fully connected hidden layers with 512 and 128 neurons, respectively, using ReLU activation functions for non-linear feature transformation. The output layer employs a softmax activation function for probabilistic multi-class classification. The network is optimized using the Adam optimizer with a learning rate of 0.001, and training is performed for a maximum of 200 epochs. To prevent overfitting, early stopping with a patience of 10 epochs based on validation loss is applied.

3.4 Proposed Hybrid Deep Feature Fusion and Stacked Classifier Network framework

Figure 1, depicted in the diagram, outlines a comprehensive multi-stage ensemble learning framework designed to effectively detect driver distraction and drowsiness. This system combines deep learning-based feature extraction with traditional machine learning classifiers, enhancing both accuracy and robustness. It consists of five key modules: preprocessing, feature extraction, feature fusion, base classification, and meta-classification. Each module serves a crucial role in improving the system’s generalization and predictive performance. By analyzing driver images, the framework accurately identifies various states related to driver attention and fatigue—such as head position, eye state, and yawning—through a structured stacking approach that leverages the benefits of deep learning with those of machine learning methods.

1.png

Figure 1. Simplified data flow of the proposed Hybrid Deep Feature Fusion and Stacked Classifier Network (HDFSNet) architecture

3.4.1 Input and preprocessing module

The system accepts input images of driver faces, rescaled to a standard spatial resolution of 224 × 224 pixels to ensure seamless compatibility with transfer learning CNNs. To enhance dataset diversity and mitigate class imbalance, real-time data augmentation is applied to generate balanced samples for each class. These enhancements include random transformations, namely rotational perturbations (±15°), horizontal image flipping, and brightness scaling (0.7–1.3), which simulate real-world variations in lighting, head tilts, and gaze directions.

Such data augmentation approaches play an essential and crucial role in maintaining the robustness of the model under varying lighting conditions, occlusions, and driver postures by exposing it to a wide spectrum of visual scenarios during training. In addition to this, the system employs automated facial landmark identification via Dlib’s 68-point algorithm, which ensures consistent and precise localization of key facial regions, eyes, mouth, and head pose, even when the driver’s posture varies or partial occlusion occurs.

This preprocessing pipeline, combined with the subsequent fusion of various feature representations from three complementary CNN architectures and a stacked ensemble of classifiers, significantly enhances the system’s resilience and generalization capabilities. Consequently, the model reliably detects driver drowsiness and distraction under diverse and challenging real-world driving conditions.

• VGG16: Uses stacked 3 × 3 convolutions to capture fine textures, ideal for detecting subtle eye and facial changes indicating drowsiness or distraction.
• ResNet50: Utilizes residual connections to extract deeper semantic features, helping distinguish visually similar driver states like head orientations.
• InceptionV3: Applies multi-branch convolutions for multi-scale analysis, capturing complex facial poses and cognitive load indicators.

Each model generates a high-dimensional feature vector offering a unique perspective, and these complementary features are fused for robust and accurate driver behavior classification.

3.4.2 Feature fusion module

After extracting feature vectors independently from VGG16, ResNet50, and InceptionV3, the Feature Fusion Module integrates these diverse representations into a unified feature vector for each image. Specifically, the outputs from each CNN—corresponding to the training and validation sets—are concatenated along the feature dimension using numpy.concatenate() function. This horizontal stacking guarantees the preservation and collective utilization of each model’s complementary qualities.

• VGG16 contributes low-level texture and edge information.
• ResNet50 provides mid- to high-level semantic features due to its deep residual structure.
• InceptionV3 offers multi-scale contextual insights through its parallel convolutional filters.

The resulting fused feature vector captures a rich blend of spatial, contextual, and semantic attributes, significantly enhancing the system’s capacity to differentiate between various driver behaviors. This multi-perspective representation is critical for robust and generalized classification in real-world scenarios.

The deep feature vectors extracted independently from VGG16, ResNet50, and InceptionV3 are subjected to L2 normalization prior to fusion to ensure consistent scaling across heterogeneous feature spaces and to prevent any single backbone from dominating the fused representation due to differences in feature magnitude. In the primary experimental configuration, no dimensionality reduction is applied, as preserving the full high-dimensional feature space is critical for retaining fine-grained discriminative cues associated with subtle driver behaviors such as marginal eye closure, yawning, and head pose deviations. After normalization, the feature vectors are concatenated along the feature dimension to form a unified fused feature representation for subsequent classification.

3.4.3 Base classifiers module

The fused feature vector obtained from the feature fusion stage is fed into an ensemble of three distinct base classifiers, each designed to depict different patterns in the data and contribute uniquely to the overall decision process:

• SVM: Utilizes a polynomial kernel to map input features to higher dimensions, allowing it to find the most effective hyperplanes that distinguish classes with optimal margin—especially effective in high-dimensional feature spaces.
• MLP: A feedforward neural network comprising of one hidden layer, proficient at learning intricate and nonlinear relationships among integrated features. Enhances the model’s capability to distinguish subtle behavioral patterns.
• SGD: A linear classifier trained using the log function, well-suited for large-scale and sparse datasets owing to its computational efficiency and ability to converge quickly.

Each base learner generates class probabilities as intermediate outputs, forming the foundation for the next phase of classification handled by the meta-learner.

3.4.4 Meta classifier module

To improve classification performance, the system uses a stacking ensemble with logistic regression as a meta-classifier. It takes the prediction probabilities from three base classifiers—SVM, MLP, and SGD—as input features. This meta-classifier learns the best way to integrate these outputs to:

• Integrate the strengths of each base model.
• Reduce individual biases or weaknesses.
• Enhance generalization and robustness on new data.

This stacked approach provides a more balanced and accurate prediction, especially for complex multiclass driver behavior classification.

For the stacking ensemble, each base classifier (SVM, SGD, and MLP) outputs a class probability vector for every input sample rather than a discrete class label. These probability vectors are concatenated to construct a meta-feature vector of dimension 3 × N, where N denotes the total number of predefined target classes.

A logistic regression meta-classifier is trained on these meta-features using a five-fold cross-validation protocol on the training set to minimize information leakage and decrease overfitting. The regularization parameter is set to C = 1.0, and the lbfgs solver is employed for stable multi-class optimization. This probabilistic stacking approach enables the meta-classifier to learn an optimal combination of the complementary decision patterns produced by the heterogeneous base classifiers, thereby improving robustness and generalization under real-world driving conditions.

3.4.5 Output prediction classes

The meta-classifier’s final output identifies eight driver behavior states, divided into two groups: distraction and drowsiness. The distraction states — BottomCenter, MiddleCenter, MiddleLeft, and MiddleRight—indicate different head poses and gaze directions to assess driver attention. The states of drowsiness, closed eyes, open eyes, no yawn, and yawn reflect key physiological signs such as eye closure and yawning. Together, these predictions provide a comprehensive analysis of driver alertness and focus, enabling timely detection of risky driving behavior.

3.5 Dataset description

The study used a merged dataset of 41,094 labeled images across eight classes: four head poses (BottomCenter, MiddleCenter, MiddleLeft, MiddleRight), two eye states (Open, Closed), and two yawning statuses (yawn, no yawn). These classes highlight key visual signs of driver drowsiness and inattention. The dataset was divided into training (70%, 28,509 images), validation (15%, 6,160), and testing (15%, 6,175) sets to support effective model training and evaluation. Real-time data augmentation was applied to each subset. This diverse and well-structured dataset enables robust hybrid models, such as HDFSNet, to accurately detect and classify driver states from visual cues. To reproduce the reported performance (≈ 99.08% accuracy), a minimum of 30,000 well-balanced and diverse images with data augmentation is recommended. This ensures sufficient coverage of real-world variations in lighting conditions, facial expressions, and head poses, supporting robust generalization of the HDFSNet model in practical deployment scenarios.

Figure 2 illustrates various driver states used for training and evaluation. The top row shows head pose variations: (a) BottomCenter, (b) MiddleCenter, (c) MiddleLeft, and (d) MiddleRight. The bottom row depicts eye and yawning states: (e) Closed, (f) Open, (g) no_yawn, and (h) yawn. These categories help to assess driver drowsiness and distraction under various conditions.

2.png

Figure 2. Typical samples from the merged dataset

The merged dataset integrates an internally curated driver monitoring dataset (head pose classes) and a public Kaggle dataset (https://www.kaggle.com/datasets/serenaraju/yawn-eye-dataset-new/data) for drowsiness-related classes. The internal dataset is not publicly distributed due to data ownership and privacy constraints. The original merged dataset contained a total of 17,789 images and exhibited significant class imbalance. To address this, offline data augmentation (±15° rotation, horizontal flipping, and brightness scaling of 0.7–1.3) was applied to underrepresented classes. After augmentation, the dataset was expanded to a total of 41,094 images, with all eight classes balanced to contain an equal number of samples. The balanced dataset was subsequently partitioned into training (70%), validation (15%), and testing (15%) subsets.

This study presents an advanced and efficacious approach to driver distraction and drowsiness detection through the proposed HDFSNet model, which leverages hybrid deep feature fusion and stacked classifiers. By combining the strengths of multiple pre-trained CNN architectures and integrating them with SVM, SGD, and MLP classifiers, the model achieves a superior accuracy of 99.08% on merged datasets. Comparative analysis with existing methods highlights its robustness and efficiency in identifying driver fatigue and inattention. The findings indicate that the suggested framework not only outperforms individual and fusion-based models but also offers a scalable solution for enhancing road safety.

This study advances driver behavior recognition by addressing three critical gaps in existing systems: limited feature representation, ineffective feature integration, and poor generalization under real-world variability. The proposed HDFSNet framework demonstrates that multi-level feature fusion across complementary CNN backbones enables the simultaneous modeling of fine-grained facial micro-expressions and global head pose context, providing a richer and more discriminative representation than single-backbone approaches. Beyond representation, this work establishes the role of decision-level classifier stacking as a generalization mechanism, where heterogeneous base classifiers learn complementary decision boundaries and a logistic regression meta-classifier optimally integrates their probabilistic outputs. This strategy consistently reduces inter-class confusion among visually similar driver states and enhances robustness to variations in illumination, occlusion, and pose. Together, these contributions provide both a theoretical and applied justification for hybrid feature fusion and stacked ensemble learning in safety-critical driver monitoring systems, demonstrating that the performance gains of HDFSNet stem from principled architectural design rather than increased model complexity alone.

Future work will focus on improving the model by incorporating larger and more diverse datasets, integrating physiological signals (EEG, heart rate, etc.), and optimizing it for real-time deployment on embedded systems. Additionally, incorporating temporal and multimodal learning techniques could further boost its reliability in real-world driving environments.