Video-Based License Plate Detection Using Deep Learning Boundary Filtering Method for License Plate Detection from Surveillance Images

In computer vision and security, the identification of vehicle license plates from surveillance footage is an essential application. First, surveillance footage is extracted into frames [1]. To handle privacy problems and guarantee responsible technology usage, ethical and legal standards are crucial [2]. It is anticipated that further technology advancements will improve license plate border detecting systems' effectiveness and dependability, eventually enhancing general security and public safety [3].

The identification of automobile license plates from surveillance footage is greatly aided by deep learning [4]. Relying on sophisticated neural network designs, including region-based models or convolutional neural networks (CNNs), makes it possible to accurately identify license plates [5]. These algorithms get sophisticated characteristics from large datasets, which improves their capacity to identify plates in a variety of scenarios. Annotated datasets comprising pictures of cars and the license plates that go with them are used for training [6]. By identifying license plate areas on its own, the deep learning algorithm eliminates the requirement for human feature engineering [7]. By using pre-trained models on huge picture datasets, transfer learning improves model efficiency even further. The ongoing progress in deep learning methodologies aids in the enhancement of license plate recognition systems, rendering them more resilient and dependable in actual surveillance situations [8]. The major contributions of the article are:

• Boundary filtering method for detecting license plate of vehicles from video inputs segregated as frames
• Employing conditional neural learning for feature filtering and similarity-based region detection to maximize the connectivity
• Performing a real-time image-based experimental analysis to verify the proposed method’s working
• Performing a comparative analysis to verify and assess the proposed method’s metric-based improvements

The article’s organization is: Section 2 presents the discussion of different methods related to number plate detection with their pros and cons. In Section 3 the proposed boundary filtering method is discussed with suitable illustrations and explanations. Section 4 presents the experimental and comparative analysis results using real time images and metrics. Section 5 concludes the article with the findings, limitations, and future work.

The process of recognizing the vehicle's license plates is initiated with the detection of textural features from the image sequence/video. The proposed BFM using CNL is applied to improve the detection accuracy of the license plate from the frame images. The vehicles are captured and detected through surveillance systems in different instances that rely on better license plate detection accuracy for their textural feature extraction. The license plate detection accuracy is a prominent factor for which the false positive is to be reduced through conventional neural networks. The three main segments such as Preprocessing of the image, number plate detection, and extraction of textural features are performed to achieve maximum accuracy. In Figure 1, the proposed method is illustrated.

The license plate boundary is detected with filtering conditions. The congruent textural features in input frame images are identified and filtered for trained inputs. Based on the image sequence analysis, the similar boundary wavelets detected in the training images help to accurately shape the license plate region from the input images. The conditions of maximum similarity and boundary displacement connectivity are observed from the extracted features and are verified throughout the training process using CNL until reach maximum precision.

1.png

Figure 1. Proposed BF Method using CNL

3.1 Preprocessing of the frame images

BFM is the model that makes use of deep learning for textural feature extraction from the input frame images. The video output is analyzed using algorithms/methods for improving the license plate detection precision. CNL is used to train the framing inputs captured from the moving vehicles for accurately identifying vehicle misbehaviors on the roadside. The obtained frame images from moving vehicles using surveillance systems placed in the roadside environment. The textural features are extracted from the frame images for license plate detection and are pursued using deep learning based on the pixel arrangement and intensity in the particular region. Neural learning is used to train the frame input-output for precisely detecting the license plate through correlation from the already trained/identified frame inputs. The process of BFM is pursued to identify, where the congruent textural features are initially identified and filtered. Therefore, the first video output of the license plate $\operatorname{Vid}_o\left(\operatorname{Lics}_{p l t}\right)$ is represented as:

$\operatorname{Vid}_O=\frac{1}{T e} \| \sum_{N=1}^{T e}$ text $_{\text {fex }}\left(\right.$ Lics $\left._{\text {plt }}\right)-F P \|$                    (1a)

where,

text $_{\text {fex }}\left(\right.$ Lics $\left._{\text {plt }}\right)=\frac{1}{\sqrt{\pi}} \sum_{N=1}^{T e=0} \frac{\text { Licp }_{\text {boundy }}}{\text { Cong }_{t f}} T e$                 (1b)

And,

$F P=\frac{1}{\sqrt{\pi}} \sum_{-\infty}^{\infty} \frac{x_i(T e)+y_i(T e)}{T e}$            (1c)

where, text $_{\text {fex }}\left(\right.$ Lics $\left._{\text {plt }}\right)$ and $F P$ denotes the textural features of the Lics ${ }_{p l t}$ and false positives were identified from the frame images. If $i$ and $L i c p_{\text {boundy }}$ denotes the number of license plate images captured from the current instance and the license plate boundary detected. The textural features are extracted for processing the images based on varying wavelet transform in both x and y coordinates. In this method, the CNL with filtering conditions is used to precisely varying wavelet transform in both x and y coordinates. In this method, the CNL with filtering conditions is used to precisely detect the license plate boundary. If $x$ and $y$ represent the rising and falling edges identified from the frame images at random time intervalsTe. Hence, the images are analyzed for the conditions $x \in[0, \infty]$ and $y \in[-\infty, 0]$ such that the following textural feature extraction equation of the filter is expressed as:

$\begin{aligned} y_N\left[\operatorname{Cong}_{t f}\right]= & 0.5 y_N\left[\operatorname{Cong}_{t f}-1\right] -0.5 y_N\left[\operatorname{Cong}_{t f}-2\right] +x\left[\operatorname{Cong}_{t f}\right]-x\left[\operatorname{Cong}_{t f}-1\right]\end{aligned}$        (2)

Based on the filtering condition, the initial congruent textural features are filtered and suppressed based on trained frame inputs from the stored dataset. The feature extraction process is illustrated in Figure 2.

2.png

Figure 2. Feature extraction process illustration

In the above Figure 2, the frames with $i$ inputs are identified to extract $x$ and $y$ under Licp boundy. Such extractions are validated for each boundary detected in vid $\left(\right.$ Lics $\left._{\text {plt }}\right)$ and text $_{\text {fex }}\left(\right.$ Lics $\left._{\text {plt }}\right)$ are differentiated. The 0.5 -based even distribution for $x$ and $y_N$ are used to detect Licp boundy . Therefore, the normalization process is reliable to improve similarity-based verification. The feature extracted is used to correlate FP across Lics $_{p l t}$ to improve $y_N$. For alltext $t_{\text {fex }}\left(\right.$ Lics $\left._{p l t}\right)+F P$ illustrates a complete sequence of license plate video output analysis for $x$ and $y$ wavelet at different time intervals is expressed as $(N \times T e)$. Here, the variable $N$ indicates the total filtering process of congruent textural features from the instance. Filtering condition is used to reduce the false positives that occur in $\operatorname{Vid}_o\left(\operatorname{Lics}_{p l t}\right)$. False Positives take place in frame inputs due to false boundaries detected at the time of video output analysis in random time intervals. This normalization follows the filtering condition that is described by:

$\left.\begin{array}{rl}\text { Normalization }\left(\text { text }_{\text {fex }}\left(\text { Lics }_{\text {plt }}\right), T e\right)= & \frac{\operatorname{Licp}_{\text {boundy }}(T e)}{\operatorname{Cong}_{\text {tf }}} * \frac{N}{2} S M L_{\text {high }}\left(x_i-y_i\right) \\\text { Normalization }(F P, T e)= & \frac{\text { Te }\left(x_i+y_i\right)}{T e} * \frac{N}{2} S M L_{\text {low }}\left(x_i-y_i\right)\end{array}\right\} $             (3)

where,

$\left.\begin{array}{c}S M L_{\text {high }}=N(T e)\left|\frac{\alpha+\beta}{2}\right| x(T e)_{i-1} \\ \text { and, } \\ S M L_{\text {low }}=N(T e)^{-1}\left|\frac{\alpha+\beta}{2}\right| y(T e)_{i-1}\end{array}\right\}$          (4)

As per the Eqs. (3) and (4), the variables $S M L_{\text {high }}$ and $S M L_{\text {low }}$ represent the high and low similarity based on already trained frame inputs from the dataset. The factor $N(T e)$ and $N(T e)^{-1}$ is the congruent and low textural feature filtering condition based on $S M L_{\text {high }}$ and $S M L_{\text {low }}$ addressed from the inputs. Based on the detection of the license plate boundary (i.e.) $x$ or $y$, the high/low similarity textural features are used to shape the region using the frame inputs. The variables $\alpha$ and $\beta$ represent the training inputs used for the license plate boundary detection. Hence, the normalized wavelet based on $\operatorname{Vid}_o\left(\operatorname{Lics}_{p l t}\right)$ is described as:

$\left.\begin{array}{r}\operatorname{Vid}_o[N(T e)]=\frac{2^{\frac{\alpha+\beta}{2}}[(i \times T e)-N]}{S M L_{\text {high }}-S M L_{\text {low }}} \\ \text { and } \\ \operatorname{Vid}_O[N(T e)]=\frac{2^{\frac{\alpha+\beta}{2}}}{T e}\left[\sum_0^{\infty} \frac{S M L_{\text {high }}[(i \times T e)-N]}{F P}-\sum_{-\infty}^0 \frac{\left.S M L_{\text {low }}[n \times t)-N\right]}{F P}\right]\end{array}\right\}$            (5)

In Eq. (5), the normalized false positive less video output is identified after performing the filtering condition. From this $\operatorname{Vid}_o[N(T e)]$, the two factors namely similarity and boundary displacement are extracted for further analysis. Eqs. (6a) and (6b) used to validate the similarity (Similar(Te)) and boundary displacement $\left(\right.$ Boundry $\left._D(T e)\right)$ in different time interval is as follows:

$\begin{gathered}\operatorname{Similar}(T e)=\frac{1}{2 \pi(N \times T e)}\left|\sum_{N=1}^{\alpha, \beta}\left(x_i-y_i\right) F P^{i-1}\right|, \forall i \in \alpha+\beta\end{gathered}$             (6a)

And,

$\begin{aligned} \operatorname{Boundry}_D(T e)= & -\sum_{i=\emptyset_l}^{\emptyset_h} \operatorname{Similar}(T e) =\log \operatorname{Similar}(T e)_i\end{aligned}$          (6b)

From the above equation, $\emptyset_h$ and $\emptyset_l$ are the high and low similar boundary indices observed from trained frame inputs. The connectivity of maximum $\operatorname{Similar}(T e)$ and Boundry $_D(T e)$ for the $N(T e)$ as in Eq. (6c):

$\operatorname{conect}[N(T e)]=\frac{\text { Boundry }_D(T e)}{\log \left[\frac{\operatorname{Vid}_O\left(\text { Lics }_{\text {plt }}\right)}{\emptyset_h-\emptyset_l}\right]}$                 (6c)

3.png

Figure 3. Neural training model for congruency analysis

This connectivity is verified for synchronizing maximum similarity and boundary displacement alone with the different time intervals for precise detection. The classification of maximum and minimum $Similar(T e)$ and $ Boundry_D(T e)$  is performed through the deep learning process. This classification helps to identify the true positives and false positives in both features. In this classification, the textural features are independently analyzed at each level of the neural network followed by classification output. The neural training model is illustrated in Figure 3.

In Figure 3, the neural learning process for $S M L_{\text {High }}$ and $S M L_{\text {Low }}$ are validated across $\emptyset_h$ and $\emptyset_l$ differentiation. This $\emptyset_h$ is equated with $T_e$ to increase the $N\left(T_e\right)$ for $\alpha$ and $\beta$. Here, $\emptyset_h$ with $T_e$ is the $\alpha$ and $\emptyset_h$ with $N\left(T_e\right)^{-1}$ is the $\beta$ under a different $y_N\left[\right.$ cong $\left._{t f}\right]$. The training iterations for congruency are analyzed from the normalized output of $\emptyset_h$ and $\emptyset_l$. If connect $\left[N\left(T_e\right)\right]$ is observed between successive high and low $(x, y)$, the Boundary $D_D$ is detected. This learning does not pursue any conditional validation and thus the $\max D$ is identified from $N\left(T_e\right)$ and $N\left(T_e\right)^{-1}$ variations (Figure 3). The frame inputs and training images are determined as in Eqs. (7a) and (7b) for the conditions of maximum similarity and boundary displacement connectivity.

$\operatorname{CNL}[\operatorname{Similar}(T e, \alpha, \beta)]=-\sum_{i=1}^N T e_i-\sum_{i=1}^{\alpha, \beta} \operatorname{conect}[N(T e)]_i-\sum_{i=1}^N \sum_{j=1}^{\alpha, \beta} T e_i \operatorname{conect}[N(T e)]_i$              (7a)

And,

$\Delta[\operatorname{Similar}(T e, \alpha, \beta)]=\frac{\left(\text { Cong }_{t f}\right)^{-\operatorname{CNL}[\operatorname{Similar(Te)]}}}{\sum_{i=1}^{N \times T e}\left(\text { Cong }_{t f}\right)^{-C N L[\operatorname{Similar}(T e)]_i}}$              (7b)

From the Eqs. (7a) and (7a), CNL[.] indicates the conditional neural learning function for similarity index identification and ∆[.] represents the initial training image at random time intervals. Similarly, the initial training image and already trained frame inputs are comparatively analyzed for final boundary displacement detection $ Boundry_D(T e)$ as:

$\begin{aligned} & C N L\left[\text { Boundry } y_D(T e), \text { Similar }(T e)\right] =\left\{\begin{array}{l}\sum_{i=1}^N T e_i \emptyset_h \frac{1}{S M L_{\text {high }}}, \text { if } \cdot x_i(T e) \in[0, \infty] \\ \sum_{i=1}^N T e_i \emptyset_l \frac{1}{S M L_{l o w}}, \text { if } \cdot y_i(T e) \notin[0, \infty]\end{array}\right.\end{aligned}$             (7c)

And,

$\Delta\left[\text { Boundry }_D(T e), \operatorname{Similar}(T e)\right]=\frac{T P+F P}{F P}$          (7d)

In the above equations, the connectivity of similarity and boundary displacement is verified throughout the process of training until maximum license plate detection precision through deep learning. The maximum precision is achieved using the condition $C N L$ [Boundry ${ }_D(T e)$, Similar $\left.(T e)\right]$ is computed for both $x$ and $y$ coordinates. This computation helps to differentiate the true positive and false positive based on time intervals to achieve the maximum possible detection. Based on the connectivity analysis, conditional neural learning is used for both C N L [.] and $\Delta$ [.] instances. The condition of maximum similarity and boundary displacement connectivity is to reduce condition-failing features. The textural features are extracted from the maximum/minimum similarity and boundary displacement are independently verified for video output analysis. The high/low similarity and boundary displacement connectivity are verified through CNL. The condition-failing features in high and low similarity observed features are administered to prevent false positives and thereby improve true positives. Conditional neural learning ensures filtered textural features between the training inputs for accurate region detection. The filtering is pursued for identifying condition-failing features in frame inputs through deep learning. The aforementioned process is discussed in the following sections. The conditional analysis of the learning process is illustrated in Figure 4.

The conditional validation based on connect $\left[N\left(T_e\right)\right]$ is performed in the above learning process. The $\emptyset_h, \emptyset_l$, and $\Delta$ [.] are the iterated process for $D$ collaboration. The $T_e$ and $\left(T_e\right)^{-1}$ variants are useful in deciding boundary split and similarity checks. In this process, the new boundary split is augmented with new $D$ for $\Delta\left[^{\prime}\right]$ validation. However, the continuous $D$ is yet to be completed from $\theta$ extracts. If $\theta$ satisfies $\left(T_e\right)$ or $\left(T_e\right)^{-1}$ similarity checks, then $D$ is correlated for detecting the boundary of the number plate. The conditional failure requires a boundary similarity check for $\emptyset_h$ or $\emptyset_l$ or both (Figure 4).

4.png

Figure 4. Conditional analysis of the learning process

3.2 False positive reduction

The conditional neural learning is defined using two types of segments namely connectivity and no-connectivity between the training images and trained frame inputs. The connectivity is responsible for achieving maximum congruent textural features and boundary displacement whereas in no-connectivity administers low similarity textural features and false positives. The connectivity is performed with $Frame _{\text {in }}=\{1,2, \ldots \theta\}$ set of frame inputs; this frame input is used to extract textural features from all the neural network layers. The Frame $_{\text {in }}$ contains different types of features in different time intervals $T e$. Let $Q$ represent the number of condition-failing features that are present in the frame inputs. Based on the verification, the number of image sequence processing per unit time $P U_{T e}$ such that the license plate detection $(Lics_d)$ is given as:

$\begin{aligned} & Lics_d  =\left\{\begin{array}{c} Frame_{\text {in }} \times P U_{T e} \times Te \forall Frame_{\text {in }}:: Te \, and \, Q=0 \\ F P \times \frac{\theta-Q}{Frame_{ in}} \times Te \forall\left(\text { Frame }_{in}, Q\right):: Te \, and \, Q \neq 0\end{array}\right.\end{aligned}$            (8)

Such that,

Frame $_{\text {in }}:: T e=\sum_{i=1}^N P U_{T e}$          (9)

And,

$\left(F r a m e_{i n}, Q\right):: T e=\sum_{i=1}^N P U_{T e}-F P \sum_{i=1}^Q T P$              (10)

where,

$F P=\frac{T P+P U_{T e}}{T P}$            (11)

Based on the above equations, the false positives and condition-failing features from frame inputs in different time intervals. In the above equation, $ Frame_{in}:: T e$ and $(Frame_{in}, Q):: T e$ represents the mapping of the maximum/minimum similarity index addressed from the video processing and false positives in the time interval Te. The license plate detection from the surveillance system is processed based on boundary displacement. In the connectivity observation, the frame input sequence and region are the added-up factors for improving the detection precision using the mapped time interval Te. For training the images, no-connectivity-based detection and filtering are performed. The detection of license plates between $\theta \in$ $Frame_{in}$ and $Q$ is computed through the observation of their connectivity and time metrics. In Eq. (8), the condition $Q>\theta$ produces condition-failing features from the frame inputs. The time metric for filtering the conditionfailing features and the routine connectivity based on $(\theta \times$ $P U_{T e}$ ) are the verifying conditions for detection.

$T e_m=\sum_{i=1}^\theta \frac{Frame_{in}-Q}{F P}$              (12)

And,

$ℸ L i c s_d=\frac{\text { Lics }_d}{(\theta-Q)}\left(\right.$ $Frame_{i n}-F P$         (13)

In the above equation, the variable $T e_m$ and $ℸ L i c s_d$ represents the timed connectivity and routine detecting instance. From Eqs. (12) and (13), the precise detection of the license plate is achieved at each instance. This analysis is performed to identify the conditions of $Q \neq 0$ and $Q=0$ in different intervals through recurrent analysis. The false positive reduction process is illustrated in Figure 5.

The FP reduction process is illustrated in the above Figure 5 using $\theta$ as the input. This method first estimates $T_{e_m}$ to verify if it requires $\alpha$ and $\beta$ or both. Depending on the verified condition, if $\tau L i c s_f$ is the required output, and then $\emptyset_h>\emptyset_l$ alone generates precise $D$. This is false positives free requiring an added training. Considering the $\left.\Delta{ }[^{\prime}\right]$ verification, new false positives are presented from causing errors in the further iterations. The case of $Q$ requires new feature verification between different $x$ and $y$ to detect $y_N$. Thus, the different processes of $S M L_{\text {High }}$ and $S M L_{\text {Low }}$ are precise in detecting new $T_{e_m}$ until the FPs are reduced. The recurrent analysis is dependent on filtering sequences from similarity and boundary displacement connectivity verification such that precise detection is achieved using deep learning. If this sequence is observed in any instance, then the license plate detection from $\theta \in Frame_{i n}$ is terminated to reduce no-connectivity observed instances between different frame orientations. The neural learning gives the output of maximum similarity to achieve precise detection to address the false positives. The boundary in the frame inputs is accurately identified and segregated to improve precision. This prevents false positives and condition-failing features by processing frame inputs whereas, the true positive is high.

5.png

Figure 5. False positive reduction process