A Novel Fast Responding Driver Assistance Technique with Efficient Lane Detection and Collision Avoidance Using Dynamic Feature Extraction in Any Environment

Here, we are proposing the method which is more accurate, time efficient and works in all environmental condition such as in poor illumination condition as well as in too bright condition. Moreover, this system works in case of missing lane markings on the road as well. The different steps involved in detecting the lane, the lane departure warning alert, frontal vehicle detection and its collision warning alert, as well, are shown in the system architecture diagram shown in Figure 2. The algorithm comprises of the few broad steps which included the image enhancement of the captured image, lane detection and lane departure warning alert. The image enhancement is basic step which is most vital and the complexity, processing time and the accuracy of the algorithm depends on it. System takes video input from camera. The captured video is divided into frames and every 3rd frame is fed to the system. Time taken by proposed system is as small as the time require to acquire two frames from the camera. After the ROI extraction, the image is passed through the proposed spatial domain model. Lane extraction is done and if there is no lane marking on the road proposed algorithm draws the imaginary lane and make decisions. In this paper, comparative study of proposed algorithm with various algorithms is given.

2.png

Figure 2. System flow diagram of the proposed architecture

Real time video or saved video V of the road map is given as an input to the system. Not all the part in the image needed for the operation. Masking M is the concept which is used to eliminate unwanted part from the image to process. After masking we get the region of interest I_ROI on which our algorithm should work.

V={i1,i2,i3,…,in} …    (1)

M={Mask of road excluding sky and corners} …    (2)

I_ROI =V[K]∩M …    (3)

I_ROI is that part of image on which further image processing algorithms performed. First of all the driving conditions such as weather, brightness and contrast has been considered as Process noise PN_i.

For the image enhancement, the spatial domain filter is used to estimate the internal state of a process. It gives only a sequence of noisy observations. We have modelled the process in accordance with the Eq. (4).

E_i = ST_i E_i-1 + CIM_i CV_i + PN_i    (4)

The intensity of the present frame and the previous frame should get matched. The present enhanced frame extracted from camera E_i is estimates by using state transition model which is applied on previous frame. State transition model applied on the previous frame is the inverse of average intensity matrix of the present frame. Then controlled input model CIM_i is applied to the control vector CV_i. The Process noise PN_i is drawn from the zero mean multivariant normal distribution from control vector CV_i to estimate the current enhanced frame.

At any time i, control vector CV_i is given as,

CV_i = OM_i + PN_i-1     (5)

Here in Eq. (5), The Histogram of the image is calculated using frequency of number of pixel intensity values. Then the maximum entropy value of the histogram graph is calculated to find observation model OM_i which gives dynamic threshold value. Pseudo code for the enhancement of the image with proposed spatial filter is written below. The time complexity of this filter is such that it takes one tenth of second to get enhanced image result.

Proposed pseudo code for the enhancement of ROI extracted image.

Divide image I_ROI into a rectangular grid r_ij(i= 1, 2, ... , N; j = 1,2, ... , M)

for each rectangle r_ij

do

Dij ← distribution of ratios I_ROI (p)/x_t-1 (p) for all

pixel p € r_ij such that:

(x_t-1 (p) ≠ 0) ^ (x_t-1 (p) < L) ^ (I(p) < L) {deal with saturated pixel}

k_t(ij) ← median of D_ij

end for

K ← median of k_t(ij) over all rectangles r_ij

for each rectangle r_ij

do

k_t(ij) ← unknown if (k_t(ij) / K >K_thr) V (k_t(ij) / K < 1 / K_thr)

end for

for each rectangle r_ij such that k_t(ij) = unknown

do

k_t(ij) ← average of K and k_t(uv), with r_uv adjacent to r_ij and k_t(uv) ≠ unknown

end for

for all pixels p

do

let r₁, r₂, r₃, r₄ be the 4 rectangles closest to p

let w_1, w_2, w_3, w₄ be the distance from p to the rectangles centers

k_t(p) ←{ $\sum_{h=1}^{4} \mathrm{kt}(\mathrm{h}) \mathrm{wh}$} / $\sum_{h=1}^{4} \mathrm{wh}$

end for

The output of the enhanced image is then fed directly to the lane detection algorithm. If in case lane is not detected then algorithm for drawing virtual lane will get activated. µ is the factor which takes nature of the extracted lane from present and past frames. If lane is present in next frame then the equation of lane in next frame becomes equation of lane in present frame and equation of lane in present frame becomes equation of lane in past frame. But if lane is absent in next frame then by using extrapolation factor µ, equation of lane is predicted.

µ=(φ(M) – φ(E))/(φ ́(E))

where,

φ(M) is the equation of lane detected in present frame;

φ(E) is the equation of lane detected in past frame;

φ ́(E) is the derivative of equation of lane detected in past frame. µ ranges from (-) 1 to (+) 1.

(-) µ - For Left lane prediction

(+) µ - For right lane prediction

In Geometry: µ varies from (-) 90° to (+) 90°

For the lane marking detection equations from 2.6 to 2.10 is used.

$I_{R O I} \in\left\{R_{m}, B_{m}, G_{m}\right\}$    (6)

$R_{m}=\hat{R}$    (7)

$B_{m}=\widehat{B}$    (8)

$G_{m}=\widehat{G}$    (9)

$\mu=\max \left(R_{m}, B_{m}, G_{m}\right)-\min \left(R_{m}, B_{m}, G_{m}\right)$    (10)

Impact $=[0,2,4]$    (11)

$I(H)=\sum_{i=0}^{\text {length }(I)} \frac{P[i]-P[i+1]}{\mu}+\left.(\operatorname{impact}[i] 2)\right|_{P=\text { length }[I]-\text { comoress rate }+1 \cdots \cdot} \quad \quad \quad$    (12)

$R O I(s)=\frac{\mu}{\max \left(R_{m}, B_{m}, G_{m}\right)}, \mid R O I(s)=0 \equiv R O I(s)=\infty$    (13)

$R O I(V)=\max \left(R_{m}, B_{m}, G_{m}\right)$    (14)

$W t=\{$ Filter for white colour intensity $\}$    (15)

$Y t=\{$ Filter for yellow colour intensity $\}$    (16)

Let W_m be extracted white marking,

${{w}_{m}}\left[ i,j \right]=\sum\limits_{m=\frac{width\text{ }of\text{ }{{w}_{m}}}{2}}^{\frac{width\text{ }of\text{ }{{w}_{m}}}{2}}{\sum\limits_{n=\frac{heigth\text{ }of\text{ }{{w}_{m}}}{2}}^{\frac{heigth\text{ }of\text{ }{{w}_{m}}}{2}}{{}}}$

$w_{t}[m, n] . R O I_{(H S V)}[i-m, j-n]$    (17)

$w_{t}[m, n]$ and $Y[m, n]$ is the kernel for white mask and yellow mask respectively.

Convolution of these kernels with ROI extracted images gives coordinated of lane markings. If lane is missing the then proposed mathematical model is used to draw virtual marking of lane.

Suppose we have previously observed lane s and v is the missing lane. As s and v is the vector with same direction. Then by using vector addition rules we get x, which is combinations of the vectors drawn from the previous detected lanes. This is used to derive all possible combination vectors in a superset or database stored in the memory to be used to estimate the virtual lane in subsequent steps. Then by using vector addition rules we get,

$x=s+v$    (18)

To get all the possible values of the lanes within threshold limit let covariance matrix be $R_{v}$,

$R_{v}=E\left\{v v^{H}\right\}$    (19)

where, $v^{H}$ denotes the conjugate transpose of v. The y is the possible combinations of the vectors drawn from the previous detected lane.

$y=\sum_{K=-\infty}^{\infty}h^{*}[k] x[k]=h^{H} x=h^{H} s+h^{H} v=y_{s}+y_{v}$    (20)

By observing all the previous lane marking which is stored in memory the estimated virtual line is given as

$\mathbf{V}_{L 1}=\frac{\left|y_{s}\right| \dot{2}}{E\left\{\left|y_{v}\right| \dot{2}\right\}}$.

$\mathbf{V}_{\mathrm{L} 1}=\frac{\left|h{^{H}}s\right|^{2}}{E\left\{\left|h^{H} v\right|^{2}\right\}}$    (21)

$\mathrm{E}\left\{|\left. h^{H} v\right|^{2}=E\left\{\left(h^{H} v\right)^{H}\right\}=h^{H} E\left\{v v^{H}\right\} \mathrm{h}=h^{H} R_{V} h\right.$.    (22)

The resultant values of the estemated virtual lane with is stated as,

$\mathrm{V}_{\mathrm{L}}=\frac{\left.\left(R_{v^{1 / 2} \quad h}\right)^{H}\left(R_{v} 1 / 2 \quad s\right)\right| \quad ^{\wedge} 2}{\left(R_{v} 1 / 2 \quad h\right)^{H}\left(R_{v} 1 / 2 \quad h\right)}$    (23)

$\mathbf{V}_{\mathrm{L}} \frac{\left|\left(R v^{1 / 2} \quad h\right)^{\wedge} H\left(R v^{1 / 2} \quad s\right)\right| \quad ^{\wedge} 2}{\left(R v^{1 / 2} \quad h\right)^{\wedge} H\left(R^{1 / 2} \quad h\right)} \leq \pi r^{2} \frac{\left[\left(R v^{1 / 2} \quad h\right)^{\wedge} H\left(R^{1 / 2} \quad h\right)\right]\left[\left(R^{-1 / 2} \quad s\right)^{\wedge} H\left(R^{-1 / 2} \quad s\right)\right]}{\left(R v^{1 / 2} \quad h\right)^{\wedge} H\left(R^{1 / 2} \quad h\right)}$    (24)

$\mathrm{V}_{\mathrm{L}}=\frac{\left|\left(\left(R v^{1 / 2} \quad h\right)^{2}\right)\left(R v^{1 / 2} \quad s\right)\right|^{2}}{\left(R v^{1 / 2} \quad h\right)^{\wedge} H\left(R v^{1 / 2} \quad h\right)} \leq s^{H} R v^{-1} \quad s \ldots$    (25)

$R^{1 / 2} \quad h=a R^{-1 / 2} \quad s$    (26)

$\mathrm{VL}=\frac{\left|\left(R_{v}^{\frac{1}{2}} \quad h\right)^{H}\left(R_{v}^{-\frac{1}{2}} \quad s\right)\right|^{2}}{\left(R_{v}^{\frac{1}{2}} \quad h\right)^{H}\left(R_{v}^{\frac{1}{2}} \quad h\right)}=\frac{\alpha^{2}\left|\left(R_{v}^{\frac{-1}{2}} \quad s\right)^{H}\left(R_{v}^{-\frac{1}{2}} \quad s\right)\right|^{2}}{\alpha^{2}\left(R_{v}^{\frac{-1}{2}} \quad s\right)^{H}\left(R_{v}^{\frac{-1}{2}} \quad s\right)}=\frac{\left|s^{H} R_{v}^{-1} \quad s\right|^{2}}{s^{H} R_{v}^{-1} s}=s^{H} R_{v}^{-1} s .$    (27)

Let α be arbitrary real number.

$\mathrm{h}=\mathrm{a} R v^{-1} \mathrm{~s}$    (28)

$\mathrm{E}=\left\{\left|y_{v}\right|^{2}\right\}=1$    (29)

$\mathrm{E}\left\{\left|y_{v}\right|^{2}\right\}=a^{2} s^{H} R_{v}^{-1} \mathrm{~s}=1$    (30)

$\alpha=\frac{1}{\sqrt{s^{\mathrm{H}} R_{v}^{-1} s}}$    (31)

The virtual missing lane is then estimated by using Eq. (27) with the minimum mean square error. This Error equation for proposed system in between imaginary drawn line and actual line is given by Eq. (32).

|E_n| ≤ fⁿ⁺¹ (z) |x-a|ⁿ⁺¹ / (n+1)!…    (32)

where, En is the error, x is the actual value, a is the centered polynomial and n is the degree of the polynomial.

Once the lane is detected or virtual lane is drawn then the lane departure warning is to set by using practically calculated threshold. The vanishing point and the mid-point of the vehicle is calculated and then by using Pythagoras theorem Euclidean distance ɗ_l is calculated. The threshold values are dynamic and based on the road geometry, width of the total road captured. In this experiment, to make reading real time for nay road geometry ɗl / ɗ ratio is considered. The P1 and P2 are the threshold values set for left and right departure respectively. If the departure measure is less than 0.25 (P1), left departure warning is issued. If the departure measure is greater than 0.75 (P2), right departure warning is issued. If the vehicle is within the lane, departure measure shows the value between 0.25 and 0.75 and no warning is generated. Also, if ɗ (Euclidean distance between midpoint of right lane and left lane) is less than the threshold limit D, set as value 0.5, then the ‘Lane Crossing’ warning will be issued. Following is the illustration of Lane departure condition:

mou_hao_hou_.png

The lane width is calculated from the two lane midpoints mp1 and mp2 of left and right lanes. Euclidean distance is calculated as the square root of the sum of the squared differences between the two vectors. Euclidean distance is useful when we need to calculate the distance between two rows (or lane here) of numerical values. Hence we have chosen Euclidean distance rather than other measures like Hamming distance, Manhattan distance.

mou_hao_hou_1.png

Euclidean distance between Hough origin and the left lane midpoint is calculated as per equation:

ɗl= $\sqrt{(H o-m p l x) ^{2}+(H o-m p l y) 2}$

ɗl= $\left\|m p l-H_{o}\right\|=\sqrt{\left\|H_{o}\right\|^{2}+\|m p l\|^{2}-2\left\|H_{o} \cdot m p l\right\|}$

ɗl vector is estimated in a given ROI in such a way that, for a Euclidean space E, the two vectors ɗl and ζ1 follows the equation:

$\lambda_{l}, ς_{l} \in E$

$\left\|\lambda_{l}+ς_{l}\right\|=\left\|\lambda_{l}\right\|^{2}+\left\|ς_{l}\right\|^{2}$

where, ζ1 is the line segment identified using Hough transform on the left lane.

Similarly, the vector ɗr, Euclidean distance between Hough origin and the right lane midpoint is calculated.

ɗ_l as Euclidean distance between midpoint of left lane and Hough origin

ɗ_r as Euclidean distance between midpoint of right lane and Hough origin

ɗ as Euclidean distance between midpoint of right lane and left lane

Lane departure measure-: Β (ɗ_l - ɗ_r)

If Β (ɗ_l - ɗ_r) < threshold P1 and ӓ greater than the threshold ɗ then, Left Departure warning is issued to the driver.

If Β (ɗ_l - ɗ_r) > threshold P2 and ӓ greater than the threshold ɗ then, Right Departure warning is issued to the driver. Conditions for lane departure and crossing are shown in Table 1.

Table 1. Conditions for lane departure and lane crossing situations

‘Lane Cross or Danger’ warning is issued to the driver indicating the threat of an accident, if the driver ignores departure warnings and the vehicle is beginning to actually change the lane. Hence, two threshold values are used to generate the warning message based on drift of vehicle towards the left or right of the road.

For the frontal vehicle detection technique, the cascade classifier is used to detect vehicle. Cascade classifier consists of a collection of different stages, and each stage worked on weak learner image part. Positive and negative images are passed through cascade object detector function which results in cascade classifier and then original image is passed through this classifier to detect frontal vehicle. The algorithm which is used here is as follow.

Algorithm: Vehicle detection algorithm:

1. Pick fmax (maximum acceptable false positive rate per layer)

2. Pick fmin (minimum acceptable false positive rate per layer)

3. Let’s FR target is target overall false positive rate

4. Let’s Ep is a set of positive example

5. Let’s En is a set of negative example

6. Let’s FR0= 1, D0 = 1, and i = 0

{FP₀: overall false positive rate at layer 0, D0: acceptable detection rate at layer 0, i: the current layer}

7. While FRi > Ftarget {FP_i: overall false positive rate at layer i}

     A.    i++ {layer increasing by 1}

     B.     ni = 0; FP_i= FP_i-1 {ni: negative example i}

     C.     While FP_i > f* FP_i -1

a. ni ++ {check a next negative example}

b. Use Ep and En to train with xml

c. Check the result of new classifier for FRi and D0

d. Decrease threshold for new classifier to adjust detection rate,

r>=d*FP_i -1

     D.     En = empty

     E.      If FP_i > Ftarget, use the current classifier and false

detection to set En

A new novel cost effective method of lane detection, departure warning and collision detection is proposed. Also, the algorithm which is used for the proposed system takes care of all challenging environmental situations like, too bright or poor light conditions, rainy condition, on curvature, with missing lane markings at different speed conditions of the vehicle.

The propose system uses the mathematical model along with cut section of the road image for extraction of area of interest. By using a new concept, recursive method and the lane marking masking methods are used in order to detect whether the vehicle is going away either right side or left side from the center of the lane and raise the warning. In case of possibility of collision, the system computes the distance from the frontal vehicle and gives the warning. Our system has maximum accuracy as compared with other methods. When we compare the response time then Hough transform takes slightly less time than our proposed framework but its error rate is higher than our system. In case of error rate Gabor transform gives best result but it has very high response time. Thus by taking consideration of all these factors one can easily conclude that proposed framework gives much better result compare to other algorithms. From all these aspects, the proposed system architecture is successfully verified with real videos and images under almost all possible conditions.