An Image Recognition Algorithm of Bolt Loss in Underground Pipelines Based on Local Binary Pattern Operator

In modern cities, an important aspect of urban planning is the construction of underground pipelines. However, the underground conditions in urban areas are very complex, and the available underground space is extremely limited. It is difficult to detect the damages of underground pipelines or the loss of relevant accessories in time.

At present, the damages of underground pipelines are mainly detected in two stages: the image of the target pipeline is captured by a robot, and then evaluated by professional inspectors.  By this strategy, lots of pipeline images need to be analyzed manually, which limits the accuracy and timeliness of detection. To overcome the defects, this paper proposes an image detection algorithm of bolt loss in underground pipelines. Using image recognition technology, the proposed algorithm can automatically detect the missing position of bolts on the target pipeline.

Image recognition technology [1] replaces humans with computers to process and recognize such signals as words and images, and complete the tasks of classification and identification. This technology features advanced intelligence similar to human vision. Figure 1 provides the basic flow of image recognition.

As shown in Figure 1, image recognition covers three critical steps: preprocessing, feature extraction, and pattern classification. Specifically, preprocessing filters and reduces the noise and interference signal from the original image, and transforms the brightness or color of the image, improving its visual quality and effect. Feature extraction mainly segments the region of interest (ROI), extracts the features or special information, and identifies the key features that are most conducive to classification. Pattern classification selects a suitable classifier to determine the class of the image, according to the key features. Therefore, this paper attempts to improve the image recognition performance of the proposed algorithm from the three aspects of preprocessing, feature extraction, and pattern classification.

1.png

Figure 1. The basic flow of image recognition

Feature extraction is a prerequisite for image recognition. In the field of image recognition, the LBP-based extraction of texture features is a research hotspot. However, the existing LBP operators need to be further improved. In this section, the encoding and pattern selection method of LBP operator are improved to extract eigenvectors with strong discriminability, and eliminate the interference of useless patterns.

3.1 Theory on LBP operator

The LBP operator is a feature extractor that describes the texture distribution in the local region. Taking the grayscale of the central pixel in the local region as the benchmark, the LBP operator differentiates the other pixels in the local region, encodes the direction of the local difference, statistically analyzes the codes, and extracts the local texture features of the image.

In general, the LBP operator adopts the 3×3 neighborhood centering on the current pixel as the template for encoding, and the grayscale of the current central pixel as the threshold. Along the specified direction, the central pixel is compared with the eight pixels in its neighborhood in turn. The comparison results are binarized and assigned different weights. Finally, the cumulative value of the coding is added up with the LBP code value of the current pixel. Through point-wise scanning of the entire image, an LBP response image can be obtained. The histogram of this image is called the LBP feature.

To adapt to the texture feature analysis with different image sizes and sampling intensities, Ojala et al. [19] extended the 3×3 neighborhood to a circular neighborhood with different radii and number of sampling points. In this way, the LBP operator is modified into LBP_C,R:

$L B P_{C, R}=\sum_{i=0}^{C-1} u\left(g_{i}-g_{o}\right) 2^{i}$     (1)

where, g_o is the grayscale of the central pixel in the neighborhood; g_i is the grayscale of C sampling points evenly distributed on the circle with $g_{o}$ as the center and R as the radius; u(g_i-g_o)=1 if $g_{i} \geq g_{o}$, and u(g_i-g_o)=0 if otherwise; 2ⁱ is the weight of each sampling point in the neighborhood, which ensures the uniqueness of LBP_C,R encoding of the current g_o.

By the definition of LBP_C,R, the C pixels in the local neighborhood can form 2^C different outputs. For a region of ordinary size, the LBP operator can generate various patterns, making the final histogram so sparse as to be statistically insignificant. Pattern selection is necessary to reduce the pattern redundancy of the LBP operator, without sacrificing the ability of texture description.

For the circular neighborhood, as long as the value of u(g_i-g_o) is not all 0 or 1, the code value of the LBP will change with the rotation angle of the image. As the circular neighborhood rotates about the current pixel, a series of different LBP code values can be obtained, of which the minimum is taken as the final LBP code of the current pixel:

$L B P_{C, R}^{t^{i}}=\min \left\{f_{R O}\left(L B P_{C, R}, i\right), i=0, \ldots, C-1\right\}$     (2)

where, f_RO() is the rotation function, by which the neighborhood pixels of the current LBP operator rotates clockwise for i times.

It can be inferred from the encoding process of $L B P_{C, R}^{t^{i}}$ operator that the code value is invariant to rotation, but poorly discriminable. The image is very likely to contain some LBP patterns, while unlikely to have some other patterns. The most probable patterns contain most of the texture features. For them, there are at most two variations from 0 to 1 or from 1 to 0 in the binary encoding of the circular neighborhood. This kind of LBP patterns is defined as the uniform pattern of the LBP:

$L B P_{C, R}^{U}=\sum_{i=1}^{C}\left|u\left(g_{i}-g_{o}\right)-u\left(g_{i-1}-g_{o}\right)\right|$     (3)

Under different brightness levels, the LBP operator has strong discriminability and robustness. During the extraction of texture features, however, the operator only encodes the direction of the local difference, and ignores the texture features in terms of amplitude.

As shown in Figure 2, the same LBP code value was obtained through differential comparison and binarization for two local neighborhoods, which lie at different positions and differ sharply in texture feature. This indicates that the LBP operator loses part of the texture information during the extraction of texture features.

2.png

Figure 2. The loss of texture information

Therefore, the encoding method of the basic LBP operator was improved by comparing the mean grayscales and mean differential amplitudes of the local neighborhood and the global image.

3.2 Improved LBP operator

3.2.1 Encoding strategy

The encoding strategy is a three-stage process: the possible texture directions in the local neighborhood are grouped, the spatial template of each possible direction is used to differentiate the central pixel from each local pixel, and the direction and amplitude of the difference are included in the encoding process.

In the local neighborhood, there are three possible texture directions: edge texture, rectangular texture and oblique angle texture. For each texture, four spatial templates are needed to calculate the corresponding LBP code value.

For edge texture encoding, the LBP code value can be calculated by:

$\begin{aligned}

L B P_{L 1}^{C D}=\sum_{i=0}^{C / 2-1}(&\left.u\left(g_{i}-g_{0}\right) \odot u\left(g_{o}-g_{i+C / 2}\right)\right) 2^{2 i} \\

&+u\left(\left(g_{i}-g_{o}\right)\right.\\

&\left.\left.-\left(g_{o}-g_{i+c / 2}\right)\right) 2^{2 i+1}\right)

\end{aligned}$     (4)

where, $u\left(g_{i}-g_{o}\right) \odot u\left(g_{o}-g_{i+c / 2}\right)$ is the possible directions of edge texture, i.e. 0°, 45°, 90°, and 135°; $u\left(\left(g_{i}-g_{o}\right)-\left(g_{o}-g_{i+c / 2}\right)\right)$ is the trend of grayscale intensity in the current texture direction.

For rectangular texture encoding, the LBP code value can be calculated by:

$\begin{aligned}

L B P_{L 2}^{C D}=\sum_{i=0}^{C / 2-1} &\left(\left(u\left(g_{2 i-1}-g_{o}\right) \odot u\left(g_{o}-g_{2 i+1}\right)\right) 2^{2 i}\right.\\

&+u\left(\left(g_{2 i-1}-g_{o}\right)\right.\\

&\left.\left.-\left(g_{o}-g_{2 i+1}\right)\right) 2^{2 i+1}\right)

\end{aligned}$     (5)

where, $u\left(g_{2 i-1}-g_{o}\right) \odot u\left(g_{o}-g_{2 i+1}\right)$ is the possible directions of rectangular texture; $u\left(\left(g_{2 i-1}-g_{o}\right)-\left(g_{o}-g_{2 i+1}\right)\right)$ is the trend of grayscale intensity in the current texture direction.

For oblique angle texture encoding, the LBP code value can be calculated by:

$\begin{array}{c}

L B P_{L 3}^{C D}=\sum_{i=0}^{C / 2-1}\left(\left(u\left(g_{2 i}-g_{o}\right) \odot u\left(g_{0}-g_{2 i+2}\right)\right) 2^{2 i}\right. \\

+u\left(\left(g_{2 i}-g_{o}\right)\right. \\

\left.\left.-\left(g_{o}-g_{2 i+2}\right)\right) 2^{2 i+1}\right)

\end{array}$     (6)

where, $u\left(g_{2 i}-g_{o}\right) \odot u\left(g_{o}-g_{2 i+2}\right)$ is the possible directions of oblique angle texture; $u\left(\left(g_{2 i}-g_{o}\right)-\left(g_{o}-g_{2 i+2}\right)\right)$ is the trend of grayscale intensity in the current texture direction.

The traditional LBP operator only emphasizes on the texture information in the local neighborhood, failing to consider the correlation and difference between local and global grayscales. Here, the comparison between local and global grayscales is added to improve the recognition performance of the LBP operator.

First, the comparison between local and global grayscales can be encoded by:

$L B P_{a v g}^{C D}=u\left(\sum_{i=0}^{C-1}\left(g_{i}+g_{o}\right) / C+1-\sum_{i=0}^{N-1} g_{i} / N\right)$     (7)

where, $\sum_{i=0}^{C-1}\left(g_{i}+g_{o}\right) / C+1$ is the mean grayscale of the (C+1) pixels in the local neighborhood of the current central pixel; $\sum_{i=0}^{N-1} g_{i} / N$ is the global mean grayscale of the image.

Next, the comparison between the mean change amplitudes of local and global grayscales can be encoded by:

$\begin{aligned}

L B P_{\operatorname{mag}}^{C D}=u\left(\sum_{i=0}^{C-1}\right.& a b x\left(g_{i}-g_{o}\right) / C \\

&\left.-\sum_{i=0}^{N-1} \sum_{i=0}^{C-1} a b s\left(g_{i}-g_{o}\right) / N \times C\right)

\end{aligned}$   (8)

where, $\sum_{i=0}^{C-1} a b x\left(g_{i}-g_{o}\right) / C$ is the mean change amplitude of the C neighborhood pixels and the central pixel in the local neighborhood after grayscale difference; $\left.\sum_{i=0}^{N-1} \sum_{i=0}^{C-1} a b s\left(g_{i}-g_{o}\right) / N \times C\right)$ is the global mean amplitude of the whole image after the grayscale difference of C-neighborhood.

Then, an improved LBP operator encoding was formed by concatenating three groups of sub-codes of local texture encodings ($L B P_{L 1}^{C D}, L B P_{L 2}^{C D}, L B P_{L 3}^{C D}$) and two groups of sub-codes of global comparison information encodings $\left(L B P_{a v g}^{C D}, L B P_{\operatorname{mag}}^{C D}\right)$. The improved LBP operator not only reflects the local features like texture direction and grayscale trend, but also contains the global features of the entire image. It can extract richer texture information, and achieve better discriminability.

The standard LBP_C,R operator can generate 2^C patterns (code values). For example, 256 patterns will be generated if C=8. Obviously, it is not suitable to directly take the output of the standard operator as the final eigenvector. Meanwhile, $L B P_{C, R}^{t^{i}}$ can reduce the total number of patterns to 36, and $L B P_{C, R}^{U}$ can lower that number to 59.

The above operators modify the encoding pattern of the LBP operator to varied degrees. Choosing an operator means accepting the corresponding modified encoding pattern. To select the most discriminable pattern class, the following factors were fully considered: the similarity and difference between images, the occurrence probabilities of different patterns, the consistency of pattern distribution, and the contribution of class information to pattern selection.

After the features of a texture image have been extracted by the LBP operator, the probability for each pattern to appear in the image was counted. The patterns with high probabilities were treated as the main patterns that illustrate image texture, while those with low probabilities were deemed as irrelevant information or noise. Next, the consistency of pattern distribution was measured, that is, the variation in the number of the same pattern appearing in different images. If the pattern distribution is relatively consistent, the pattern was regarded as the image background. On this basis, the authors designed a pattern selection strategy based on the weighted ranking of classes:

First, the original LBP patterns of the training set are grouped into different classes. Then, the occurrence probability and standard deviation of each pattern are calculated for each class. According to the standard deviation, different weights are assigned to the occurrence probabilities of different patterns. After that, the weighted occurrence probabilities are ranked in descending order. Finally, the main patterns of different classes of images are combined as the final main pattern eigenvector. The above process can be mathematically described as follows:

Step 1. Initialize a set of N training images S={X₁, X₂, …, X_N}, and the LBP pattern vector of each image X_i=(v₁, v₂, …, v_M), where v_i is the times that the LBP pattern appears in the current image.

Step 2. Set the number C of the possible classes of the current image, and categorize the image set S into C classes, such that the same images fall into the same class $S_{C_{i}}=\left\{X_{1}, X_{2}, \ldots, X_{N_{i}}\right\}$, C_i=1, 2, …, C.

Step 3. Calculate the total number of occurrences of each LBP pattern: $\text { TotalHist }_{c_{i}, i}=\sum_{X_{i=1}}^{N 1} v_{i, X_{i}} X_{i}$ , and the standard deviation of each LBP pattern: $\text { SDHist }_{c_{i}, i}=\text { StandardDeviation}\left(v_{i, X_{1}}, v_{i, X_{2}}, \ldots, v_{i, X_{N}}\right)$ .

Step 4. Assign a weight to the occurrence probability of each pattern based on the standard deviation.

Step 5. Calculate the weighted probability of each, sort the new image set $\vec{S}$ in descending order of the weighted probability, and take the top k patterns as the main patterns of the current class.

Step 7. Select the k main patterns of each class $S_{C_{i}}$ in turn by Steps 3-5, and record the $\vec{k}_{C_{1}}$ value of the selected patterns.

Step 8. For the eigenvector set S of LBP patterns of the original image, take the pattern class corresponding to k_total as the final eigenvector of main patterns.

After Gabor transform of the original image, a total of 40 response images could be obtained in different channels. Then, the LBP operator was applied for feature extraction, producing a 112-dimensional eigenvector in each channel. Obviously, the 40×112 dimensional eigenvector cannot be directly recognized by the SVM classifier. Therefore, the improved LBP operator was called to select the main patterns of the 40 channel eigenvectors. The optimal feature combination of each channel was obtained through separate recognition of each channel. Table 1 records the optimal feature retention dimension and recognition rate of each channel.

Table 1. The optimal feature retention dimension and recognition rate of each channel

As shown in Table 1, the recognition rates of different channels differed sharply. The performance of our algorithm can be greatly improved by optimizing the combination of eigenvectors.

After optimal combination of eigenvectors was determined for each channel, the SVM classifier was adopted to identify the eigenvectors of each channel, and different weights were assigned to the predicted class tags of each channel. To test the effects of the weighting process on the recognition performance, the recognition rates of our algorithm were compared with different weight optimization methods: no weight optimization, taking recognition rate as weight, adaptive boost, and the improved TLBO. The experimental results are shown in Table 2.

Table 2. The recognition rates with different weight optimization methods

As shown in Table 2, the improved TLBO led to better results than the other two weight optimization methods, in terms of search ability and recognition accuracy. The weight optimization by the improved TLBO provides a good basis for pattern classification and image recognition.