Comprehensive Analysis to Detect Optimal Vehicle Position for Roadside Traffic Surveillance Using Lightweight Contour-Based CNN

In the rapidly expanding era of technology, the systematic monitoring of individuals has become pervasive. In smart cities, the demand for surveillance extends to roadside traffic areas, where video surveillance systems are instrumental in analyzing traffic flow, detecting incidents, and enforcing traffic laws [1]. The roadside surveillance encompasses intricate trending technologies, good-quality and meaningful data sources with reliable communication protocols. However, the implementation of such type of complex setup and their regular maintenance incurs a considerable cost. This cost may occur just because of complex software, highly configured hardware, and high computation resources. Therefore, striking a balance between the advantages and drawbacks of complex surveillance systems becomes imperative in addressing the multifaceted challenges those are associated with them.

To collect the data from different road-side traffic areas the wireless-sensor network [2, 3] has appeared as the most capable technology. In addition, computer vision also plays a very crucial role in numerous surveillance applications. It is found capable of handling challenges like real-time computation amidst many vehicles at peak traffic times and occlusions [4].

In recent years, the deep-learning techniques have been harnessed during vehicle surveillance because of their good performance and features. Its adoption is substantiated due to its gainful attributes, robustness, generalization, and scalability. For the same, the Deep Learning Convolutional-Neural-Network (CNN) comprises two core components namely feature extraction and classification. Feature extraction is aimed at the acquisition of relevant features in the input data, while classification is responsible for labeling the input data based on the ascertained characteristics. This architectural configuration is illustrated in Figure 1.

The feature extractor component typically includes multiple convolutional layers, each designed to learn fundamental features such as edges and corners, which represent low-level features in the context of the data analysis process. These low-level features are then amalgamated and processed by subsequent layers to learn higher-level features such as shapes. The classifier encompasses a fully connected layer, which takes the extracted features as an input and a probability distribution over the possible vehicle classes as an output. Figure 2 depicts the working of CNN to detect the vehicle.

1.png

Figure 1. Convolutional neural network architecture

2.png

Figure 2. Working with CNN to detect vehicle and feature extraction

1.1 Smart city framework for transport development and integration

The concept of a smart city represents an advanced urban environment built on robust data infrastructure and sophisticated frameworks, specifically tailored for the development and integration of transportation systems. The foundational elements of the smart city framework include physical infrastructures, networking systems, central computing centers, and data storage systems. All elements are essential in the context of smart transport management, as illustrated in Figure 3 [5].

3.png

Figure 3. The general smart city framework

At the core of this architecture lies the physical infrastructure, strategically planned with sensors, devices, and cameras. These devices are aimed at capturing and generating transportation-related data. This raw data is then transmitted through network communication mediums to a central computation center, where it undergoes processing to extract pertinent details, such as vehicle information during Red-Light-Violation incidents at roadside zebra crossings.

The seamless exchange of data between acquisition devices and central computing centers is facilitated through various communication channels and mediums. The processed data then traverses through these communication mediums to reach virtual machines, cloud computing devices, and servers within the central computing center before being stored. The exchange of information is essential; however, the emphasis is on confirming the reliability of the data [6]. Reliable information is indispensable for making swift decisions and generating timely responses within the framework of a smart city. The applications within smart cities utilize gathered data to forecast outcomes, which depend significantly on a well-designed network and astute management systems [7]. The effectiveness solely relies on a well-designed network and an ITMS. They both are essential for the overall functioning of the smart city framework. Roadside traffic surveillance takes center stage in smart cities as an integral part of an ITMS. Video streams act as the predominant medium in surveillance with video analytics. Figure 4 depicts a comprehensive workflow of a modern surveillance system, outlining the intricacies of surveillance within a smart city.

4.png

Figure 4. A comprehensive workflow of the modern video surveillance system

The surveillance process initiates with video acquisition, where cameras and sensors are strategically deployed as acquisition devices. The acquired video data is then transmitted to a central processing unit for further analysis, where raw video streams are transformed into digital formats to enable efficient processing and storage. In the module analysis stage, the processed data is fed into dedicated software modules for object detection by employing deep-learning models for accurate results. Following data processing and analysis, the information is stored in a dedicated unit. These units are either on-site or cloud-based, catering specifically to the surveillance of roadside traffic and incorporating ITMS.

1.2 Integrated-traffic-management-systems (ITMS)

The ITMS is illustrated by a conjunction of technologies, data inputs, and communication protocols. They are aimed at enhancing the integration of modern transport development. The deployment of such a system demands meticulous planning of acquisition devices and equipment. The intention behind this plan is to consider a coverage area, camera angles, and lighting conditions. The system incorporates two distinct camera types: the Automatic Number Plate Recognition (ANPR) camera and the Red-Light Violation Detection (RLVD) camera, each serving a specific role as illustrated in Figure 5. This setup is implemented across the city and establishes a strong interconnected network for efficient transmission in capturing real-time information for transport management and integration. This setup is aimed to optimize the detection and tracking of vehicles during their stationary periods, emphasizing safety and security.

The architectural configuration illustrated in Figure 6 and Table 1 provides a comprehensive overview of the smart city ITMS architecture.

5.png

Figure 5. Camera set-up of ITMS at road-side traffic area

6.png

Figure 6. ITMS architecture in Smart-City

These acquisition devices in the combination of RLVD and ANPR are situated approximately 20 feet (6.5 meters) above the road and 13 meters from the zebra crossline, the cameras possess a focal length of 3.2-3.5 meters, strategically focused on the zebra crossline. The ANPR camera is committed to detecting the vehicle's number plate and capturing a single image, while the RLVD camera functions as a surveillance camera, capturing four additional images upon detecting a violation. The captured data is transformed through a media converter and is transmitted back to the Local Processing Unit (LPU) via fiber optic cables. The interconnection of devices is facilitated through an L-2 switch employing cat-6 cables. Subsequently, the LPU transmits this processed data to the control room via fiber optic cables, where further processing occurs. The control room is equipped with an L-3 switch, a storage unit, and a set of 3-4 servers, including the Central Server, Data Server, and Media Server. The Central Server, serving as the master server, issues instructions to the other servers, while the Media Server functions as an aggregation point for data from various junctions. It operates as a distributed system to reduce server load. Moreover, the data are stored here in multiple replicas to ensure reliability. So that if one server goes down or is manipulated during a cyberattack, the same data can be retrieved from another server.

Table 1. Elements and their uses within an ITMS architecture in Smart-City

This manuscript focuses on a cost-efficient model for extracting low-level vehicle features for efficient roadside traffic surveillance. That can be achieved with the combination of the Integrated Traffic Management System (ITMS) with Smart-City transport development. The proposed work employs a lightweight contour-based CNN to extract hierarchical information from the vehicle by employing a custom video dataset while considering real-world scenarios. The key contributions of this work include two phases:

(1) Transforming the acquired vehicle video into separate frames.

(2) Selecting the best vehicle position frame to maximize feature extraction.

The proposed work incorporated a Region Proposal Network (RPN), Region of Interest Pooling or Region of Interest Align, and Non-Maximum Suppression to optimize the entire detection process. Apart from this, the feature extraction is enhanced through the utilization of the Structural-Similarity-Index (SSIM) with thorough parameters calculation, including Mean Luminance Similarity (MLS), Mean Contrast Similarity (MCS), Mean Structure Similarity (MSS), and Peak-Signal-to-Noise-Ratio (PSNR) for each duo frame.

The use of contour-based detection algorithms in the proposed methodology is justifiable for several reasons, despite the availability of state-of-the-art. The choice of the proposed approach aligns with the specific objectives, dataset characteristics, and real-world traffic surveillance challenges like varying lighting conditions, complex scenes, and occlusions. Contour-based detection algorithms are inherently versatile for handling complex scenarios. Unlike some algorithms that rely on predefined patterns or features, contour-based methods are adaptive and can identify object edges even in challenging situations.

Overall, this research contributes to the improvement of robust surveillance solutions tailored for smart city transport, thereby enhancing shelter and efficiency in urban transport systems.

Over the preceding decades, a multitude of datasets have been extensively employed in the domain of object detection and vehicular surveillance, as comprehensively elucidated in the literature review undertaken within this research manuscript. The ensuing Table 4 provides prevalent datasets employed in the realm of vehicle detection and video surveillance, accompanied by descriptive details and rationale for their utilization.

The custom dataset excels in aligning with specific research goals for vehicle detection in realistic traffic scenarios. It offers diversity, real-world legitimacy, and a deliberate focus on the research objectives. Real-world challenges, such as partial occlusions, enhance the dataset's complexity, addressing unique hurdles during vehicle analysis. While benchmark datasets may not precisely match the targeted research focus on optimal vehicle position detection. The custom dataset stands out for its alignment with specific research goals with the ability to capture vehicles from multiple angles conducive to comprehensive vehicle analysis.

The experimental dataset in this study is meticulously curated to meet the evolving needs of integration of transport development. The target location is the roadside traffic signal area, here vehicles temporarily halt according to signal time. Typically, the signal time ranged from 30 seconds to 2 minutes, hence accordingly videos were recorded. This bespoke dataset features 20 video recordings 5 for each Indian four-wheeler vehicle, the duration of the video ranges from 50 seconds to 1.5 minutes. Each video contains 300 frames. Notably, the videos include four distinct vehicle brands: Hyundai i20, Maruti-WagonR, Maruti-Swift, and Maruti-Suzuki-Brezza, selected to accommodate variations in body length and style. All decided vehicles exhibit different colors. The reason for selecting different vehicles was to account for variations in body length and style across different models, which surely prevented the model from being biased during training.

The videos were captured in the morning, afternoon, and evening, with a moving camera. The videos were strategically recorded from diverse angles (+15 to -15 degrees relative to the scene) to ensure coverage of vehicles from various positions and viewpoints. This approach facilitates the identification of optimal vehicle positions. The videos also consider different meteorological conditions, including sunny and opaque scenarios, with some recordings featuring partial occlusions caused by shadows from trees and individuals. Sample video snippets of the recorded dataset are illustrated in Figure 8.

This comprehensive and diverse custom dataset serves as a valuable resource for training and testing vehicle detection, providing real-world challenges and scenarios that extend beyond the limitations of existing benchmark datasets.

Table 4. Commonly used datasets in vehicle detection

8.png

Figure 8. Recorded video snippets featuring various Indian vehicles captured at different times and under diverse occlusion conditions

5.1 Result

The proposed workflow is initiated by systematically processing a sequence of continuous video frames, ensuring the careful conversion of each frame into an image representation, and storing it in the 'frameimages' list. This methodical approach guarantees the seamless capture of every frame, laying the foundation for our in-depth analysis. To emphasize specific features within these frames, this manuscript explores various plot patterns. Notably, one of the simplest yet most insightful ways to visualize these features is by presenting the image itself. As illustrated in Figures 10, 11, and 12 the author adopts this approach to showcase the series of consecutive video frames as individual images, with each frame treated as an element in the 'frameimages' list.

Figure 13 illustrates the front, partial side, and total side view of the vehicle. In the continuation, Figure 14 illustrates all detected contours in the initial and final frames of the video for reference. Here, the dark-colored vehicle is parked in the shadow of a tree, which can be seen in Figure 14.

All detected contours are represented in green to show the different intensity values and gradients. In the first image, the contours are detected on the front glass, bonnet, bumper grill, license plate, and logo. In the second image, the contours are detected on the side glass and gate. Which reflects the maximum contours available in the first image. Therefore, the best vehicle position is illustrated in Figure 15.

To highlight the maximum information those are detected by the contours illustrated by the heatmap. Figure 16 displays the magnitude of the phenomenon in a 2-dimensional Heatmap.

In addition, Figure 17 depicts the contours of the white Maruti Swift, which has been parked on sunny times without any shade.

14.png

Figure 14. Contours detected in the front and side perspectives of an image featuring a Maruti WagonR, taken under the shadow of a tree

15.png

Figure 15. Optimal vehicle position, which has maximum features

16.png

Figure 16. Heatmap to display the magnitude of the phenomenon in 2-dimensional Heatmap

18.png

Figure 17. Detected contours in the frontal view of a White Maruti Swift Vehicle

In Figure 17, all contours are detected on the headlight, bonnet, license plate, grill, and logo. Moreover, the contours are also highlighting the image that is present on the bonnet. All this information may be beneficial for effective surveillance and maximum information gain.

Additionally, the histogram plot can shed light on how the pixel values in an image are distributed. Video histogram analysis focuses on the quantitative analysis and visualization of pixel intensity distributions within images and video frames. The analysis can provide details regarding the brightness, contrast, and pixel intensity as a whole. A histogram is generated using the flattened pixel intensity values. It is divided into 256 bins, representing the full range of pixel intensities (0 to 255). The following is a representation of the histogram plot used for illustrating the distribution of features:

Assume that X = {x₁, x₂, ..., x_n}^T is a collection of n observations of a particular feature. The feature values xi can be discrete or continuous. Let B = [B₁, B₂, ..., B_k] be the collection of k non-overlapping bins. Each of [Bj] symbolizes a range of feature values. The observations in each bin are then counted in the next step.

Let C = "c₁, c₂, ..., c_n" be the collection of counts, with c_j denoting the number of observations that fit into the bin B_j. To determine the relative frequencies in each bin, the counts must be normalized in the last step. By dividing each count c_j by the sum of the observations n and the width of the bin B_j, this is often accomplished.

Corresponding to the range and distribution of the feature values, each bin can be either uniform or varied. These are possible representations for the normalized counts or relative frequencies as shown in Eq. (7):

$f j=\frac{c i}{(n * B j)}$     (7)

Here, in Eq. (7) the fj stands for the relative frequency of observations in the bin Bj.

The relative frequencies fj can be plotted against the bin centers or boundaries to get the histogram. Depending on the desired level of granularity in the plot, the midpoints or ends of the bins can be chosen as the bin centers or limits. The resulting plot shows the distribution of the feature values graphically and can be used to spot trends or abnormalities in the data. Figure 18 portrays the pixel distribution of 2 different frames of video first and last frame sequentially, whereas Figure 19 illustrates the intensity of pixels in the video.

This manuscript also acknowledges the inevitable diversity in pixel intensities across the extracted frames. To examine the relationship between pixel intensity fluctuations and the corresponding video content, it becomes essential to calculate the average intensity across all frames, effectively representing the mean video intensity. Hence, subgraphs (a) and (b) of Figure 20 depict the average intensity of 100 frames through a line plot (showing frame-time correlation) and a scatter plot (facilitating the comparison of pixel intensities) for video analytics.

17.png

Figure 18. Histogram of pixel distribution

19.png

Figure 19. Pixel intensity in video

20.png

Figure 20. (a) Line plot and (b) Scatter plot for visualizing video intensity across 100 frames to observe the correlation between pixel intensity changes and corresponding video

In the realm of practical real-life scenarios, the two different vehicles can look the same or the same vehicle can look different in different positions, illumination, and light intensity. Therefore, metrics that replicate this behavior tend to be more effective. The Structural Similarity Index (SSIM) [67] and Scale-Invariant Feature Transform (SIFT) are two methodologies for extracting features to compare between two images. SSIM serves as a metric to evaluate the similarity between two images. The SSIM computed via Eq. (8) and its parameter descriptions are listed in Table 6.

$\begin{aligned} & \operatorname{SSIM}(I 1, I 2) =\frac{((2 * \mu I 2 * \mu I 2+\text { Const } 1) *(2 \sigma I 1 I 2+\text { Const } 2))}{\left(\left(\mu I 1^2+\mu I 2^2+\text { Const } 1\right) *\left(\sigma I 1^2+\sigma I 2^2+\text { Const } 2\right)\right)}\end{aligned}$             (8)

Table 6. Description of Eq. (8) parameters

Table 7 displays Mean SSIM (MSSIM) and Table 8 displays various parameters of SSIM including Mean Luminance Similarity (MLS), Mean Contrast Similarity (MCS), Mean Structure Similarity (MSS), and Peak-Signal-to-Noise-Ratio (PSNR) for pairs of frames (from frame 0 to 299). However, considering the overall similarity of nearly consecutive frames, only 15 comparisons were conducted between the initial and final frames, gradually transitioning between them to compute all parameters. The value ranges from -1 to +1, where +1 signifies identical or extremely similar images, and -1 indicates highly divergent images.

In Table 7, the Frame Image pair column contains frame combinations 0 and 298 i.e. first and almost the last frame. As we move from the first frame to the last frame, similar frames start coming closer. As a result, it can be understood that those frames are identical. Therefore, it can be seen in the table that the value of SSIM for frames 0-298 is small while for frames 14-284 it is large. Therefore, if we continue increasing like this, we will get a large value for the same frame, which will show their similarity.

Table 7. Mean structural similarity index (MSSIM)

Table 8. Structural Similarity Index (SSIM) parameters

In Table 8, the large value of PSNR represents high similarity and the small value represents low similarity.

Subgraphs (a) and (b) of Figure 21 illustrate the model's performance by evaluating the running accuracy and loss. Additionally, Figure 22 illustrates the minimum loss and validation loss. The model was ultimately saved with a minimum loss of 0.0816 and an accuracy of 0.9736.

22.png

Figure 21. Loss and accuracy progression throughout each iteration in every epoch

23.png

Figure 22. Minimum loss and validation loss sequentially in each epoch

The performance of the proposed work in terms of processing time elapsed/step with validation loss and validation accuracy is depicted in Table 9 and Table 10. It can be observed that as epoch increases elapsed time/step decreases, and the model ultimately achieves its optimum accuracy at iteration 2.

In addition to the performance showcased in the preceding figures and tables, the author assessed the effectiveness of the proposed approach against ten alternative methods. The comparative analysis of accuracy with these ten approaches is presented in Table 11 and Figure 23 revealing that the performance of the proposed work surpasses that of the other ten methods.

Table 9. Performance in terms of processing time per step with loss and accuracy in each epoch during Iteration 1

Table 10. Performance in terms of processing time per step with loss and accuracy in each epoch during Iteration 2

Table 11. Comparison in terms of accuracy with the other ten approaches

24.png

Figure 23. Performance of proposed work as compared to other approaches

5.2 Discussion

The use of a contour-based detection approach in the proposed methodology is justifiable for several reasons, despite the availability of state-of-the-art object detection algorithms. The choice of the proposed approach aligns with the specific objectives, dataset characteristics, and the challenges addressed in the research, particularly in the context of handling varying lighting conditions. The algorithms may exhibit sensitivity to certain conditions, adaptability, robustness to complex scenes, ability to handle occlusions, and computational efficiency. Moreover, they serve as a foundational step, paving the way for subsequent more specialized vehicle analyses and monitoring.

During the study, the author observed that the proposed approach is well suited for four-wheelers such as cars. Its accuracy on commercial vehicles, heavy vehicles such as trucks, and three-wheelers such as autos may vary because of their shape and size. Another challenge arises with identical vehicles, as in some cases, the commercial vehicle belongs to the same company, which is why they may be identical. Moreover, since a brand-new vehicle with the same company, same brand, and same color is identical, this is also a challenge.

Table 12 provides a detailed justification for this choice to show the performance of the proposed work under different scenarios.

Table 12. Justification of contour-based detection approach to perform under different scenarios

In roadside traffic area surveillance, the positioning of vehicles within the camera frame is of utmost significance. Vehicle detection in roadside traffic monitoring videos plays a vital role in ITMS. To set up video surveillance at roadside traffic, locations of cameras, and other equipment, considering factors such as coverage area, camera angles, and lighting conditions matter. The widespread equipment of surveillance cameras has resulted in a vast database of traffic footage for analysis. Conventional surveillance is often hindered by distortions resulting from camera angles and lightning. However, when the cameras are positioned at a prominent viewing angle, the road appears more distant, which affects the size of the detected objects. Effectively addressing these challenges and finding solutions is essential, especially in complex camera scenes, to enable further practical applications. To tackle this issue, contour-based detection methods are utilized. These techniques concentrate on delineating the silhouette of vehicles, encompassing their distinctive features.

In summary, by addressing these challenges and finding the optimal vehicle position, this work not only advances the field of roadside traffic surveillance but also aligns with the broader goal of optimizing the smart city framework for enhanced functionality and contributes to the overall efficiency of smart city traffic management systems. These advancements not only address the specific challenges of road-side traffic surveillance but also contribute to the broader goal of creating more intelligent, efficient, and secure urban environments.