Deep Learning-Driven Regulation of Vehicle Speed Limits in Response to Weather Conditions

1.1 Problem definition

In this study, it is aimed to automatically perform the speed control according to the weather conditions, which is done manually in a few European countries, and it is tried to be suitable for the purpose of the control together with the detection of violations. With electronic monitoring systems, it is aimed to support 24/7 inspection by using less personnel. Thanks to the computer program using image processing methods and deep learning methods, it is aimed to demonstrate that electronic inspections can be made with camera images that can be accessed remotely, without the need for high-tech and costly cameras. In addition, valuable data have been created to be used in different studies with the number and density of vehicles.

With the technological developments in electronic systems, speed limit application is widely used in electronic control systems for vehicles on highways, such as license plate recognition, speed limit control, incorrect lane change control, phone use control while driving, etc. With the increase in the availability and functionality of these systems, drivers try to ensure safer travel by complying with the rules in a conscious and controlled manner.

However, there are problems in the design and use of these electronic systems. While high-tech cameras are sufficient in some regions for speed limit enforcement, in some regions additional speed sensor devices are used due to old technology, lack of infrastructure, faulty planning, etc. The existence of these devices both increases the cost and reduces workforce efficiency as it requires maintenance.

Another issue is that in many European countries, speed limits on highways are determined based on the vehicle type and the highway used. When determining speed limits, weather conditions, which are as important a parameter as the vehicle type and the highway used, are not taken into account. In this case, driving at a speed appropriate to seasonal conditions depends on the driver's discretion and the presence of field supervisory personnel.

Over speed is the speed at which the speed limit exceeded. While speeding is the cause of the majority of fatal accidents, it is also an important factor determines the severity of the accidents. As can be seen in Figure 1 [1], the risk of accident increases as the changes in average speed increase.

Nevertheless, drivers tend to over speed while driving. According to the research conducted in European countries within the scope of the Safe Road Trains for the Environment project [2], 82% of drivers know that excessive speed is an important accident factor, yet more than 70% of them state that they exceed the speed limit. With speed limits not set in a weather-adaptive manner, the risk of accidents will depend on the discretion of drivers found to be over speeding at 70%.

In France a variable speed limit is applied according to the weather conditions. In case of rain and snow or in heavy traffic conditions, the speed limit on highways is reduced from 130 km/h to 110 km/h, and from 90 km/h to 80 km/h on city roads. In case of fog the speed limit is reduced to 50 km/h. As shown in Figure 2 [2], this method takes place manually and remains within the regional scope.

1.png

Figure 1. Speed-accident risk graph

2.png

Figure 2. Manually changed speed limit in heavy traffic in France

In addition, there are NZLIMITS in New Zealand, USLIMITS in America, QLIMITS, NLIMITS, VLIMITS etc. [2] different applications from other countries. In these applications, appropriate speed limits for highway safety are determined according to different factors such as road characteristics, accident history, number of uses and activities.

1.2 Background

1.2.1 Convolutional Neural Network (CNN)

The CNN method emerged with the development of the concept of artificial intelligence, which was questioned by the Imitation Game [3] test designed by Alan Turing in 1950. The journey of developing deep learning methods commenced in 1943 with McCulloch and Pitts' study [4], which unveiled a logical model of brain functions inspired by the human nervous system. Additionally, in 2012 Krizhevsky and Sutskever's study presented during the ImageNet Large Scale Visual Recognition Challenge [5], marked a significant milestone in advancing deep learning to a foundational level.

A Sequential Convolutional Neural Network (CNN) is a deep learning architecture characterized by its sequential arrangement of layers. In a Sequential CNN, layers are stacked one after another in a linear fashion, forming a unidirectional flow of data from input to output. Typically, the architecture starts with an input layer and proceeds with a series of convolutional layers for feature extraction, interspersed with pooling layers for spatial downsampling. Following the convolutional and pooling layers, one or more fully connected layers may be added for classification or regression tasks. The simplicity and ease of implementation of Sequential CNNs, make them a popular choice for image-based applications. In Figure 3, the Sequential CNN layer architecture used in this study is given as a visual diagram.

The mathematical expression of the convolution operation [6] is as shown in Eq. (1).

$S(i, j)=(I * K)(i * j)=\sum_m \sum_n I(i+m, j+n) K(m+n)$           (1)

In the equation, S represents the convolution operation output, I represents the input, K kernel and * denotes the convolution operation.

Convolutional Layer. The convolution layer used to determine the properties of the input is the step in which the filters performing the convolution operations are scanned horizontally and vertically in the input. For example, when the image in Figure 4, which consists of 1s and 0s as input, is scanned with the CNN filter given, the output feature map to be obtained is as shown in Figure 5 [7]. Here, while the filter is moved starting from the upper left corner of the sample image, the values between the two matrices are multiplied and the results are summed. The collected results are stored in the feature map. This process is repeated for the elements in each row and column in the visual. Figure 5 shows the final processing of the sample CNN filter in the image. The value 4, which is the (3,3) element of the feature map, is obtained as a result of the following Eq. (2).

$(1 \times 1)+(1 \times 0)+(1 \times 1)+(1 \times 0)+(1 \times 1)+(0 \times 0)+(1 \times 1)+(0 \times 0)+(0 \times 1)=4$              (2)

3.png

4.png

Figure 4. Input and CNN filter

5.png

Figure 5. Filtering and result

Rectified Linear Unit Layer (ReLU). This layer, which comes after the convolution layers, is the layer where neuron outputs are rectified to become ready for other layers. Negative values in the data coming after the convolution layer are set to zero and are not used during learning. For this reason, the ReLU activation function is often preferred in multilayer neural networks. Thanks to the operations performed in this layer, deep learning accelerates even more. Figure 6 [6] contains example for commonly used activation functions. The mathematical expression of the ReLU function is given in Eq. (3).

$g(x)=\max (0, x)=\begin{aligned} & x, x \geq 0 \\ & 0, x<0\end{aligned}$               (3)

6.png

Figure 6. ReLU activation function

Pooling Layer. Pooling layer is the layer where calculations are made in the network architecture. The success of the model created in this layer is calculated and the dimensionality is desired to be reduced. In this way, the required power and features that are determined but will not be used are filtered. There are two different pooling methods: maximum and average pooling. Maximum and average pooling transactions are shown below in Figure 7 [8].

7.png

Figure 7. Max and average pooling

Fully-Connected Layer. In this layer, the input image, which passes through the convolution layer and the jointing layer several times and is in the form of a matrix, is converted into a flat vector suitable for the classification layer. Figure 8 [6] shows the process of preparing the neurons emerging after the convolution operations for the classification layer as a single and flat vector in the full connection layer.

8.png

Figure 8. Fully-connected layer

Classification Layer. This layer comes after the fully-connected layer and converts the data coming from the neurons into logical values equal to the number of objects to be classified. For example, 1024 neurons entering this layer are output as a 1024×3 weight matrix for 3 different objects. The most commonly used classifier in this layer is softmax. Softmax produces values between 0-1 for each object to be predicted, the object with the value closest to 1 is predicted as the output of the model. Figure 9 shows an example output for the classification process.

9.png

Figure 9. Classification

Softmax. It is generally used in the last layer of neural networks, where the probability distribution of n different events is calculated. Its most important advantage is that it can handle multiple classes. Its mathematical expression can be seen in Eq. (4) [6].

$Softmax\left(x_i\right)=\frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}$               (4)

In this study, deep learning is performed with a database consisting of camera images using the CNN algorithm. The most important reason for choosing CNN as the deep learning algorithm method is that the database preprocessing of the CNN algorithm is much less than other methods and the learning processes are much faster. The weather forecast was taken as the result of the Softmax algorithm as the deep learning output. The model output will be used in weather detection and model queries have been carried out at certain periods. After the weather condition was determined according to the model output, the image processing parameters to be used in the application were updated and the speed limit was determined. Vehicle speed control has been implemented according to appropriate parameters and speed limit.

1.2.2 Morphological transformations

Morphological transformation is an analysis method based on nonlinear operations related to the shape and geometry of features in an image. It is generally used to clean the areas/objects that are desired to be separated in binary images from objects outside the region of interest. During these processes, the configuration element is hovered over the input image. Various mathematical operations are performed according to the pixel value of the image and the value of the configuration element. There are examples of structuring elements in Figure 10 [9].

10.png

Figure 10. Structuring element examples

Erosion. Erosion is used to shrink and erode objects in the image. Comparison is made by moving the structuring element pixel by pixel over the image. If the center value of the configuration element corresponds to the value 1 in the visual, and if the visual has a value of 0 under the other 1 values of the configuration element, the visual pixels under the other 1 values of the configuration element are converted to 0. Erosion is used to remove unwanted noise because it reduces peaks and increases the width of 0-pixel regions in the image. Figure 11 [9] shows the image obtained when the erosion process is applied on the first image.

11.png

Figure 11. Erosion

Dilation. Dilation is used to enlarge and expand objects in the image. The structuring element is moved pixel by pixel over the image. When the configuration element coincides with a pixel with a value of 1 in the central image, the visual pixel values below the other values are changed to 1. Expansion expands the width of 1-pixel regions in the image, so it can remove negative noise but can also increase unwanted noise. Figure 12 [9] shows the image (b) obtained when the dilation process is applied on the first image (a).

12.png

Figure 12. Dilation

Opening. First Erosion and then Dilation is performed on the original image.  While the structuring element moves pixel by pixel in the image, it eliminates less valuable pixels in the image and ensures that other regions remain the same. Thus, fine lines, spots and unwanted noise in the image are eliminated. In the 3 images in Figure 13 [9] respectively; original image, 5×5 Opened image, 9x9 Opened image.

13.png

Figure 13. Opening

Closing. First Dilation and then Erosion is performed on the original image. The aim of the closing process is to eliminate the gaps by filling them. While the configuration element is moved pixel by pixel in the image, if a 1-valued pixel of the image coincides with the element center, it ensures that the low-value pixels below the other values of the element are converted to 1. In the image in Figure 14 [9], it can be seen that the recesses and gaps in the object are closed as a result of the Closing process.

14.png

Figure 14. Closing

1.2.3 Distance calculation

Euchlidean distance is a calculation of between two points. Euclidean distance between points in n-dimensional Euclidean space [10] is given in Eq. (5).

$d_{\text {euc }}$$=\sqrt{\left(p_1-q_1\right)^2+\left(p_2-q_2\right)^2+. .+\left(p_n-q_n\right)^2}$         (5)

where, p and q represent the points in the x and y plane of the coordinates for which the Euclidean distance calculation is made.

15.png

Figure 15. Traffic lines used in France

We used lane markings and on-road measurements to automatically calculate the actual distance from the camera image. For this purpose, the T'1 marking method specified in the study [11] and shown in Figure 15 was taken as basis for Camera #4. In addition, as given in Figure 16, according to the guide published by the General Directorate of Highways (KGM) for the cameras in Istanbul [12], the system was calibrated so that the lane lines and intervals on the road in the relevant camera were 3 meters each.

16.png

Figure 16. Traffic line characteristics in Turkey

1.2.4 Kalman filter

The Kalman filter [13] was published by R. E. Kalman in 1960 with the aim of extracting useful signal from noisy measurement values. R. E. Kalman used this approach to predict states based on linear dynamical systems in state space format. More generally, the Kalman filter is an algorithm that can predict the next state of a model based on data from the previous state. The biggest reason why this algorithm is called a filter is that it can separate the actual value in the measurements and the noise by optimizing it according to the predictions it makes.

In systems using sensors such as radar, laser and sonar, the Kalman filter is usually only concerned with the position of the target. In general terms, the $Z_k$ measurement vector, H (measurement matrix) and R (covariance matrix) values for tracking the moving object are expressed as in Eq. (6), Eq. (7) and Eq. (8) below [14]. Here, the $B_x$ value is a variant of the position measurement errors and the $v_k$ value is the measurement noise.

$Z_k=H_{x_{t k}}+v_k$               (6)

$H=\left(\begin{array}{ll}1 & 0\end{array}\right)$            (7)

$R=\left(B_x\right)$              (8)

1.3 Related works

Similar studies carried out before are discussed as follows:

In the study by Chowdhury et al. [15], vehicle detection study was carried out with image processing methods. In the study, vehicle detection achieved a 95.08% success rate for the daytime period and 94.10% for the nighttime period.

In the study by Biswas et al. [16], CNN architecture was used for vehicle detection and counting. An average of 96.55% success rate was achieved in the counts made at certain time periods for more than one database. As stated in this study, vehicle counting could not be performed during heavy rain and snowy weather conditions, and it was also reported that the vehicles that changed lanes were counted more than once, since the vehicle count was made according to the vehicles passing in the lane.

In the study conducted by Hsieh et al. [17], the counting efficiency was tried to be increased with shadow filtering and linearity by using image-processing methods during vehicle counting. As a result of the study, it was seen that the vehicle counting success rate was between 92.8% and 94.8% for different cameras.

In the study conducted by Gupte et al. [18], vehicle detection, tracking and classification were performed using image processing methods. As a result of this study, vehicles were detected and tracked with a success rate of 90% in 20 minutes, 15 fps video images and classified as successful with a success rate of 70%.

In the study conducted by Van Pham and Lee [19], vehicle counting was performed using image-processing methods. As a result of the study, the vehicles in a 20-minute image were detected and counted as 98% successful.

In the study conducted by Bhaskar and Yong [20], vehicle counting was performed using image-processing methods. As a result of the study, as a result of the counting made in three different videos, a successful count of 91.26% was achieved.

In the study conducted by Yaghoobi Ershadi et al. [21], vehicle detection in different densities, weather conditions, road type, light intensity and spaces was achieved with approximately 94.02% success.

In the study conducted by Tayara et al. [22], the vehicle counting study from satellite images was carried out with a model prepared using FCRN with a 90.51% success rate.

In the study by Wicaksono et al. [23], vehicle detection was performed using image processing methods and morphological transformations, and approximate vehicle speed estimation was performed. The speed estimation success rate was 87.01%, but it was determined that the system created in this study increased the error rate at high speeds.

In the study by Chu et al. [24], temperature, humidity and weather forecasts were made using SVM architecture. At the end of the study, it was seen that the weather forecast was correct with an average of 76.6%. However, among the 5 weather conditions used in the study, it was observed that the performance was below the general average of 76.6% for rainy and cloudy weather conditions.

In the study by Zhang et al. [25], weather classification was applied using Multi-Core Learning (MKL) with a database consisting of 4000 images. The overall average success rate was 71.39%. In this model, 32% of rainy images were incorrectly classified as snowy.

In the study carried out by Roser and Moosmann [26], forecasts were made for clear, light rainy and heavy rainy weather conditions with the designed SVM model and the success rate was obtained as 85.19%.

Vehicle detection algorithms were created in the study by Luo et al. [27]. As a result of the application, a 92.25% success rate was achieved for vehicle detection.

In the study conducted by Chen et al. [28], as a result of the model created using the Faster R-CNN algorithm, vehicle detection for various types of vehicles was performed with an average success rate of 75.7%.

In the application of the speed limit that changes according to the weather, the image taken from any online camera is subjected to two synchronized processes. The CNN weather model is created from the images containing various seasonal conditions obtained from the relevant camera, which is the first process. The success rates of the models created and used in the system are between 99.51% and 99.85%. According to these models, weather forecasts are made at certain periods by using the snapshots taken from the relevant camera. After estimation, hard coded speed limit and image processing parameters are changed automatically.

The second process includes image processing and speed measurements. For this process, firstly, the approximate distance for the camera from the area of interest is calculated. Afterwards, moving vehicle and speed detections are performed using image processing methods. In the system designed, the success rates in vehicle detection, counting and calculation of speed were between 76.46% and 94.67%.

As a result of the process, a csv file is created containing the entry and exit times of each vehicle and the calculated speed values. In case of exceeding the specified speed limits, the snapshot of the relevant vehicle is saved in the specified directory. With this process, vehicle count and vehicle density information can be obtained at certain periods. Figure 35 below shows, as an example, images of vehicles exceeding the speed limit in different weather conditions.

As can be seen in Table 10, the system shows satisfactory performance in vehicle detection. For example, 2256 vehicles were counted at approximately 0.9 vehicles per second during one of the busiest hours of the day in Istanbul with Camera #13. Although the system detected every vehicle in this observation, it showed at least 76.46% success, even with errors caused by incorrectly increasing the vehicle number during the passage of some (long and wide) vehicles or due to lags in the live camera image. As seen in Figure 36, when wide and long vehicles pass through the ROI area, these vehicles can cover the entire area of interest. In this case, the entry and exit time values of the center of the detected object were used when calculating the speed values.

In order to demonstrate the accuracy of the speed estimation, 5 transition were made with the test car on the Camera #13 with normal weather condition and actual speed value was compared with the speed value calculated. As a result of these comparisons, the error rate in speed calculation was found to be 2.54%. The parameters of these transitions and the velocity calculation were verified by estimating with the Kalman filter [13].

35.png

36.png

3.1 Vehicle speed estimation verification

In the designed system, the unit distance difference between the vehicle center points in the previous frame is calculated for each frame. The same calculation was performed for the Kalman filter's center point estimates of the detected vehicle in the next frame in order to verify the velocity estimation. Velocities were calculated for each frame by the ratio of the distance between the obtained two frames to the time differences between the two frames. The Kalman filter starts its estimations from the point (0,0) and continues its estimations according to the vehicle center point in the next frame. After the first few frames, the Kalman filter appeared to make predictions close to the center point.

After the planning, a constant 60 km/h transition was made on the road where Camera #13 live broadcast and weather prediction has made by model. The live video during this transition was recorded without changing the fps value and was used as the video source of the process in the designed application. In this context, speed calculation calibration is provided for the relevant camera. After this calibration, transitions were made again from the relevant cameras and speed estimation was made in the live environment. It has been determined that these estimates are close to the real values and suitable for calibration. In Figure 37, it is seen that the speed value obtained as a result of the process during the transition with the test vehicle is 60.68 km/h.

37.png

Figure 37. Speed estimation result with test car (zoomed)

In Figure 38, there is the average velocity value obtained from the Kalman filter estimates at the time of transition with the test car. While the speed value calculated as a result of the process was 60.68 km/h, the average Kalman speed for the same vehicle was 61.74 km/h.

38a.png

38b.png

Figure 38. Kalman average speed of test car

Table 11 shows the results from previous studies and their comparison with our study. As can be seen, our study can make weather prediction more successfully than other studies because it creates a deep learning model with images taken from the camera to be predicted. Since we used image processing methods instead of machine learning for vehicle detection and we perform image processing methods via CPU instead of GPU, our success rate in vehicle detection is slightly lower than other studies.

3.2 Effects of different weather conditions on results

As seen in Table 10, it has been determined that rainy, snowy and foggy weather conditions adversely affect the image processing steps used in vehicle counting and speed calculation. As seen in Figure 39, vehicle and accurate speed detection becomes difficult under heavy rain.

39.png

Figure 39. Incorrect vehicle and speed detection under heavy rain (zoomed)

3.3 Effects of camera image features on results

40.png

Figure 40. Vehicle detection with low image quality

During the development period, problems such as access problems, physical stabilization of the camera, image quality and closing the viewing angle during rain were encountered. Figure 40 shows that the system designed from any camera with very poor image quality can work for vehicle detection and counting, but the relevant camera images do not have the necessary screen quality for weather detection.

3.4 Effects of image processing parameters on results

If the morphological transformation and background subtraction parameters are not adjusted properly, the surrounding lights cause reflections on the asphalt road, especially in the evening hours. In this case, our image processing algorithm can detect the relevant light movements as cars. In order to prevent this situation, appropriate parameters must be entered for each camera.

In Figure 41, the motorcycle and its silhouette on the rainy ground are not perceived as a vehicle since their volume is small, but are perceived as a part of the other vehicle in the image processing parameter. In this case, the speed calculation of the relevant vehicle will be incorrect. To prevent this, the second value of the structuring element used in the morphological opening process was increased for rainy weather conditions of the relevant camera.

41.png

Figure 41. Incorrect image processing parameters result-1

42.png

Figure 42. Incorrect image processing parameters result-2 (zoomed)

As seen in Figure 42, if the morphological transformation and background subtraction parameters are not set appropriately, the surrounding lights cause reflections on the asphalt road, especially in the evening hours. In this case, our image processing algorithm can detect the relevant light movements as vehicles. To prevent this situation, appropriate parameters must be entered for each camera.

Figure 43 shows an example of the situation that occurs as a result of improper selection of morphological transformation parameters during the passage of large vehicles. By increasing the initial value of the structuring element used during the morphological closure process, this situation was prevented and a more accurate speed calculation could be obtained after the midpoint of the detected integrated region moved out of the red border line.

43.png

Figure 43. Incorrect image processing parameters result-3

Figure 44 shows an example of the error encountered in the number of vehicles and therefore the vehicle speed calculation as a result of numbering the vehicles passing close to each other with the same number, if the minimum value variable used in the tracking algorithm is not entered in accordance with the camera angle and large vehicle passages. This situation was prevented by entering a specific minimum tracking value for each camera.

44.png

Figure 44. Incorrect image processing parameters result-4