An Efficient Deep Learning Model Based on Driver Behaviour Detection Within CAN-BUS Signals

To ensure safe and responsible driving, drivers must remain fully attentive to their performance and diligently concentrate on the road ahead of them [1]. Unfortunately, the driver's consideration can be easily diverted by various activities, events, and surroundings, leading to distractions that can compromise safe driving practices [2]. Driver distraction may occur in a variety of ways, such as visual (when a driver's eyes are taken off the road), manual (when a driver's hands are taken off the wheel), and cognitive (when a driver loses concentration on the road) [3]. These types of distractions can significantly increase the risk of a car crash or other types of accidents.

The increasing usage of vehicles in modern times has resulted in undesirable consequences, such as traffic congestion, accidents, injuries, fatalities, and financial losses. A survey conducted by the National Highway Traffic Safety Administration (NHTSA) and Virginia Tech Transportation Institute (VTTI) revealed that a minimum of 80% of accidents and 65% of near-accidents that take place every day on roads and highways are associated with distractions or poor decision-making while driver driving on roads and highways [4, 5]. Additionally, secondary tasks such as visual, auditory, and haptic-related tasks, like using vehicle displays, listening to the radio, or using a smartphone, can still affect driving safety [6].

The concept of driver behaviour is intricate and pertains to the Driver's ability to manage and control the vehicle in a specific environment and context for each driving scenario [7]. The Driver plays a crucial role in ensuring driving safety as they manage and manoeuvre the vehicle through a set of actions.

Research and technology advancements have led to significant developments in driver behavior detection. Various approaches are being explored, including computer vision-based methods, sensor-based techniques, biometric data analysis, and machine learning algorithms [8-15]. Computer vision techniques employ cameras and video data to track driver behavior, facial expressions, and eye movements, enabling detection of distractions and fatigue. However, concerns regarding privacy violations and occasional failures in specific situations have been raised despite notable progress in computer vision and camera-based human identification.

 However, combining these technologies can help us get closer to our objective of creating a reliable end to end system for tracking the risk of driver distraction in commonplace circumstances [16]. An offering method for tracking drivers' actions as well as carrying out modern technology and advances in artificial intelligence (AI) gives us various capabilities that can significantly expand our potential in this area [17]. Improved CAN-BUS and electronics have also been developed for analysis, real-time monitoring, reporting, and command over potentially distracting automated decision-making by applying deep learning to big datasets generated via CAN-BUS sensor.

To achieve the highest accuracy with the smallest time window size and the least amount of data, we propose an in-vehicle sensor based system for the detection of aggressive driving in real-time behaviours using Convolutional Neural Networks (CNN) combined with Recurrent Neural Networks (RNN/LSTM). The following contributions assist the paper in achieving its main goal.

• An efficient deep-learning model is proposed to detect aggressive Driver behaviour utilizing in-vehicle data from CAN-BUS sensor. The architecture we have proposed achieves better performance than existing state-of-the-art methods. It demonstrates improved accuracy, effective memory use, and reduced computational complexity.

• Our proposed framework for driver behaviour can achieve classification with a shorter time-series window as compared to prior research.

• Our study specifically examines the impact of anomaly data presence and window size, which refers to the number of time steps and the level of overlap achieved through sliding windows, on the accuracy of the results.

This section presents an overview of the data collection process from in-vehicle sensors, followed by data processing techniques and the use of deep learning models for driver behavior classification. The subsequent evaluation and validation procedures were employed to ensure the reliability and accuracy of the models, while a separate analysis of robustness was conducted to assess their performance in the presence of anomalies.

3.1 Data description

3.1.1 Participants

A cohort of 4 full-time male drivers was recruited for this study. The drivers in the study had an age range of 20 to 40 years, with an average age of 26.3 years and a standard deviation of 7.3 years. Their driving experience ranged from 4 to 20 years, with an average of 7 years and a standard deviation of 7 years. To minimize systematic errors, naturalistic driving experiments were conducted using identical equipment for all participants.

During the testing phase, real-world driving data was collected to analyze each driver's response to more than four ride requests throughout a single day of journeys. To ensure uniformity, all drivers were instructed to utilize the vehicle between 1 p.m. and 4 p.m. and were exposed to identical weather and road conditions. Prior to participation, informed consent was obtained from all participants, and measures were taken to prioritize driver safety and maintain the confidentiality of their data.

1.jpg

Figure 1. Driver route loop

The data acquisition experiment took place in an urban environment that encompassed a variety of road types. These included express lanes, arterial roads, continuous curved roads, one roundabout, and six signalized intersections. The chosen location provided a diverse range of driving scenarios for data collection. The specific road types and their spatial arrangement within the urban area are visually depicted in Figure 1. This comprehensive selection of road configurations ensures the inclusion of various driving scenarios, allowing for a comprehensive analysis of driver behavior and performance within an urban context. the participants were exposed to identical weather conditions during the testing period. While the specific details of the weather conditions are not mentioned in the provided information, measures were taken to ensure uniformity among participants.

3.1.2 Design and installation of the data acquisition system

Data collection involved the utilization of three distinct sensor types to gather relevant information. These sensors encompassed the measurement of the accelerator pedal value, throttle position signal, coolant temperature, long-term fuel trim bank 1, and short-term fuel trim bank 1.

Additionally, RPM, speed derived from the OBD-II UART sensor, yaw angle ascertained from the MPU5060 sensor, and vehicle location acquired from the Adafruit GPS breakout were recorded. By incorporating these sensor technologies, a comprehensive dataset was compiled, facilitating the analysis of various vehicle parameters and characteristics. Figure 2 illustrates the physical circuit setup for the in-vehicle sensors, including the OBD-II UART sensor, Adafruit GPS breakout, and MPU5060 sensor. This schematic representation showcases the interconnection and positioning of these sensors within the vehicle. The circuit design ensures the accurate and reliable acquisition of data from these sensors, enabling the collection of essential information for the study.

These sensors were configured to record data at a sampling rate of 1Hz, where the average value of all the data within one second was considered as the representative value for that particular second. Figure 3 provides a visual representation of this data sampling process.

3.1.3 Normalization of features and sample structure

Data preparation is essential of deep learning pipeline, a two-step process of standardization was employed. The first step involved mean subtraction, where the mean value of the data was subtracted from each data point. This process helps center the data around zero, making it easier for the model to learn and converge efficiently. The second step involved division by the standard deviation, which scales the data to have a unit variance. By dividing each data point by the standard deviation, the data was normalized and brought to a common scale, minimizing the impact of varying magnitudes among different features. This standardization process ensures that each feature contributes equally during the training phase, preventing any particular feature from dominating the learning process. By undergoing this two-step data preparation process, the deep learning pipeline was able to handle the input data effectively and improve the model's performance and generalization capabilities.

2.png

Figure 2. Physical circuit of in-vehicle sensors

3.png

Figure 3. Visual representation of collected data

$\mu=\frac{1}{M} \sum_1^M d_i$                (1)

$\sigma=\sqrt{\frac{\sum_1^M\left(d_i-\mu\right)}{M}}$                  (2)

$d_{standardized}=\frac{d_{\text {actual }}-\mu}{\sigma}$            (3)

Eq. (3) outlines the process of standardization, where each data point ($d_{actual}$) is subtracted by the mean (μ) and divided by the standard deviation (σ). This standardization process ensures that the data is transformed to have a mean of 0 and a standard deviation of 1, enabling fair comparisons and analysis across the dataset.

A sliding window approach with a step size of 10 seconds and an overlap window of 9 seconds was applied to the in-vehicle sensor data collected from the OBD-II UART sensor, Adafruit GPS breakout, and MPU5060 sensor. This sliding window technique facilitated the segmentation of the continuous sensor data into smaller, temporally overlapping windows for analysis. By setting the step size to 1 seconds and the overlap window to 9 seconds, a significant portion of the data was shared between adjacent windows, ensuring a comprehensive coverage of the sensor readings as shown in Figure 4. This approach allowed for the extraction of temporal patterns and trends in the sensor data, enabling the investigation of dynamic changes in vehicle performance, location, and orientation over time. It also facilitated the application of various time-series analysis techniques to derive meaningful insights from the collected sensor data.

4.png

Figure 4. Overlapping window sliding

3.2 Classification model

Driving datasets can be understood as collections of sequential data points arranged in a time series structure. These time series problems being classified into two categories according to the number of variables involved: univariate time series, which focus on a single variable, and multivariate time series, which incorporate multiple variables. In the context of driving datasets, it is common to have at least four time series features such as speed, acceleration, etc. This suggests that a time series with multiple variables classification issue can be used to frame the challenge of driver categorization. In the subsequent stage of our methodology, the dataset was transformed into a 3-dimintion array structure (features, samples, steps), where features denote the features number, samples represent the samples number in the dataset, and steps corresponds to window size.

To enhance the classification of time series data, Karim et al. [43] introduced hybrid deep learning model including Fully Convolution network with Long short-term memory (FCN-LSTM) network, as illustrated in Figure 5. By employing the FCN-LSTM architecture, the researchers achieved remarkable improvements and enhanced accuracy across a variety of time series classification challenges with minimum data preprocessing. The FCN-LSTM model combines the strengths of Convolutional Neural Networks (CNNs), which are renowned for their effectiveness in time series classification, with Long Short-Term Memory (LSTM) networks to capture temporal dependencies within the sequences.

In this study, we employing driving data along with FCN-LSTM to classify drivers. Without any previous feature engineering, such as the use of statistical characteristics, the data was simply entered into the model. We can attain an ideal preprocessing time using this approach. Each of these methods, as shown in Figure 5, uses the same data input, but from several angles. The data is delivered to the CNN block as a univariate time series with several time steps. k This change is made to lessen the problem of the data rapidly becoming overfit.

According to Figure 5, the FCN-LSTM architecture incorporates both a fully convolutional block and an LSTM block with dropout, as described in the study [43]. The fully convolutional block comprises three successive temporal convolutional blocks with filter widths of 128, 256, and 128, respectively, following the convolution block structure proposed by Wang et al. [44]. Each convolutional block consists of batch normalization [45], followed by a temporal convolutional layer with a momentum of 0.99 and an epsilon value of 0.001. A ReLU activation function is applied after the convolutional layer. Subsequently, a global average pooling operation is performed after the last convolution block.

5.png

Figure 5. FCN-LSTM proposed model

Concurrently, the time series input undergoes a dimension shuffle layer to reorganize its structure. The resulting transformed time series is subsequently fed into the LSTM block, which consist of LSTM layer. A dropout layer is applied after the LSTM block. The output from the global pooling layer and the LSTM block is merged through concatenation and then forwarded to a softmax classification layer.

Long short-term memory networks (LSTMs) have emerged as an advancement over conventional recurrent neural networks, addressing the challenge of vanishing gradients. LSTMs tackle this problem by incorporating memory cells and gating mechanisms that effectively regulate the storage and retrieval of past states. In each time step, an LSTM maintains a hidden vector (h) and a memory vector (m), which play a crucial role in controlling state updates and generating outputs.

The data is divided into 10-time increments, where each increment represents a time series of 10 seconds. The architecture utilizes a dual-branch structure to process the data.

In the first branch, the data is processed through the CNN block. The CNN block treats the data as a univariate time series with multiple time steps.

In the second branch, the data undergoes a dimension shuffle operation before entering the LSTM block. This operation reshapes the data, allowing the LSTM block to interpret it as a multivariate time series. The dimension shuffle layer rearranges the data to be perceived as 10 variables with a single time step. This transformation enables the LSTM block to effectively model temporal dependencies and capture the dynamics of driver behavior over time.

The outputs from both branches, representing the spatial features from the CNN block and the temporal patterns from the reshaped data in the LSTM block, are concatenated. This concatenation integrates the spatial and temporal information in a seamless manner. By combining these components, the FCN-LSTM architecture captures both local spatial features and global temporal dependencies, resulting in a more comprehensive understanding and accurate prediction of driver behavior.