Automatic Recognition for IoT Supervision Images Based on Modal Decomposition

Considering the industrial IoT system architecture, this paper divides the IoT supervision system into three parts according to its functional division: IoT supervision terminal, data transmission system and IoT remote monitoring cloud platform. Figure 1 shows the overall framework of the IoT supervision system.

1.png

Figure 1. The overall framework of the IoT supervision system

Dynamic feature extraction of monitoring moving targets is a basic task of automatic recognition of IoT supervision images, which can be achieved by effectively separating foreground and background of IoT monitoring video streams. To reduce the impact of dynamic changes in illumination, background and foreground in real application scenes on the recognition effect, a variety of methods have been proposed by domestic and foreign scholars, but most of them are unable to process the local dynamic information in the spatiotemporal data of IoT supervision. Meanwhile, in the practical application world, automatic recognition of targets will become difficult if there is no complete background frame in the IoT supervision images with complex environment. Therefore, this paper conducts a study on automatic recognition for IoT supervision images based on modal decomposition. For the problems of poor real-time performance and few samples that commonly exist in video stream target recognition, the paper proposes a dynamic modal decomposition-based feature extraction algorithm for IoT supervision video stream to build a suitable platform for IoT supervision image foreground segmentation.

The dimensionality of video surveillance image data is very high in many applications. This paper uses a dynamic modal decomposition method to perform a spatiotemporal decomposition of IoT supervision image sequences, so as to achieve dimensionality reduction of the image data, which can be specifically achieved based on a data-driven decomposition of the Koopman operator spectrum. If the A pixel matrix characterising IoT supervision images is L × M and L >> 1, it is difficult to compute the eigenvectors of A^(m) (A^(m-1) )⁺ . The eigenvectors of Ẋ_s can be computed by first decomposing them into singular values and retaining only the first s(<< L) orders, which is more computationally efficient to handle. Assuming that the approximate matrix is represented by X and Ẋ_s, we had:

$\left(A^{(m-1)}\right)=V_s \sum_s U_s^{+}$      (1)

$\dot{X}_s=V_s^{+} \sum_s U_s$      (2)

X and Ẋ_s are L × L and s × s respectively, with the first s eigenvectors being the same. Assume that the feature vectors of Ẋ_s are Ẋ_s =QΓ_s Q⁺. The feature vectors corresponding to the normal and abnormal conditions of different detection targets can be classified based on the feature values in Γ_r.

To extract features of IoT supervision images with complex environments and without complete background frames, this paper optimizes the traditional dynamic modal decomposition method, which is based on dictionary learning to extract the most essential dynamic features of IoT supervision video streams and reduce the interference of information less relevant to the monitoring target on the automatic recognition effect. Figure 2 gives the IoT supervision image foreground extraction process. A good dictionary has a sufficiently sparse model. Figure 3 shows the principle of background and foreground separation of IoT supervision images. Assuming that the input IoT supervision video image of dimension c_p is represented by a, the dictionary model of dimension c_p * L is represented by C, and the sparse matrix of dimension L is represented by β, we had:

$a=C \cdot \beta$      (3)

Assuming that the open square after the square of each component vector of β is represented by ||β||, the purpose of the IoT supervision image feature extraction is to find min||β||.

2.png

Figure 2. IoT supervision image foreground extraction process

3.png

Figure 3. Principle of background and foreground separation of IoT supervision images

The problem of constructing the dictionary model C is approximated as a bi-objective optimization problem with the objectives of ensuring that C and β_i can reconstruct a_i as distortion-free as possible and ensuring that x_i is as sparse as possible.

$\underset{C,{{\beta }_{i}}}{\mathop{min}}\,\sum\limits_{i=1}^{M}{\left\| {{a}_{i}}-C\cdot {{\beta }_{i}} \right\|_{2}^{2}}+\mu \sum\limits_{i=1}^{M}{{{\left\| {{\beta }_{i}} \right\|}_{1}}}$       (4)

The m image samples are randomly selected from the IoT supervision image sample set A as the initial samples for the dictionary model C, and β is set to 0.

The solution to a_i is discussed in this paper for a sample of IoT supervision images. The sample is assumed to be an a-vector and the sparse code to be a β-vector. If a and C are known, solving for β and needing it to be as sparse as possible means that the fewer non-zero elements of the matrix are required, the better.

Assume that C is represented by [c1, c2, c3, c4, c5] containing five matrix elements. This paper first finds the closest element to a. Assuming that it is c₄, then it derives β = [0, 0, 0, d4, 0], where the size of d₄ characterizes the weight of the matrix elements. Assuming that a = d₄ * c₄, the value of d₄ can be obtained by calculation. Based on the result of the calculation, find the residual vector a' = a - d₃ * d3 and stop the algorithm when a' is less than the pre-set threshold, and go to the next step if a' is greater than the pre-set threshold.

Calculate the closest distance to a' in c₁, c₂, c₃, c₄, c₅. If assuming it is c₂, then update x = [0, d₂, 0, d4, 0]. Assume a = d₂ *c₂ +d₄ *c₄, then update the residual vector a' = a - d₂ *c₂ - d₄ *c₄ based on the result of the d₂ calculation. Determine if it is less than the threshold, and if not, continue to find the coefficient of the next closest element. The number of coefficients of the solved elements can also be specified as a constant, for example 3, that would represent 3 iterations. Once all the x_ihave been found, C can be updated.

Coefficient matrix Y₁={ῶ¹_i,1, ῶ¹_i,2, ..., ῶ}¹_i,P^t_i=1 and Y₂={ῶ²_i,2, ῶ²_i,2, ..., ῶ}²_i,P-^t_i=1, (Y₁,Y₂∈R^M×(P-1)) are all approximated IoT supervision image sequence block coefficient matrices W₁={w_i,1, w_i,2, ..., w}_i,P-1^t_i=1 and W₂={w_i,2, w_i,3, ..., w}_i,P^t_i=1, (W₁, W₂∈R^M×(P-1)), which are obtained through dictionary learning training.

Assuming that the column vector containing the overlapping patches T is denoted by {.}^t_i=1, the number of rows of the frame sequence matrix and the coefficient matrix is denoted by M and L respectively, the colour block along all frame sequences is denoted by W={w_i,j}^t_i=1(W∈R^M×P), the regularization parameters used to control the sparsity in the coefficient matrices Y₁ and Y₂ are denoted by μ₁ and μ₂, and the coefficient matrix approximation can be obtained by solving the minimization problem shown in the following equation:

$\begin{align}  & \tilde{\omega }_{i,j}^{1}=\underset{\omega _{i,j}^{1}}{\mathop{argmax}}\,{{\left\| {{w}_{i,j}}-C\omega _{i,j}^{1} \right\|}^{2}}+{{\mu }_{1}}{{\left\| \omega _{i,j}^{1} \right\|}_{1}} \\ & \left( i=1,2,...,T,j=1,2,...,P-1 \right) \\\end{align}$            (5)

$\begin{align}  & \tilde{\omega }_{i,j}^{2}=\underset{\omega _{i,\left( j-1 \right)}^{2}}{\mathop{argmax}}\,{{\left\| {{w}_{i,j}}-C\omega _{i,\left( j-1 \right)}^{2} \right\|}^{2}}+{{\mu }_{2}}{{\left\| \omega _{i,\left( j-1 \right)}^{2} \right\|}_{1}} \\ & \left( i=1,2,...,T,j=1,2,...,P \right) \\\end{align}$           (6)

By considering the coefficient matrices Y₁ and Y₂ as the solution of the basis functions, this is similar to the extended dynamic modal decomposition method.

Suppose that a set of dynamic modes of IoT supervision images is represented by ρ = {ρ₁, ..., ρ₂} and the corresponding feature values are represented by Γ={Γ₁, ..., Γ_s}. Based on both ρ and Γ, this paper reconstructs the IoT supervision image sequences, and the number of feature vectors used is denoted by s. ρ = {ρ₁, ..., ρ₂} denotes that in the IoT supervision video streams, the monitoring target with changes at the time point p $\in$ {0, 1, 2, ..., P-1} has an associated continuous time frequency, i.e.:

${{\theta }_{j}}=\frac{log\left( {{\Gamma }_{j}} \right)}{\Gamma p}$            (7)

Suppose that the column vector of the j-th dynamic mode containing the spatial structure information is represented by ρ_j and the initial amplitude of the corresponding dynamic mode decomposition method mode is represented by β_j. Then the approximate video frames of different frequency modes at any time point can be reconstructed in the following way:

$Y\left( p \right)\approx \sum\nolimits_{j=1}^{s}{{{\rho }_{j}}{{o}^{\theta _{j}^{p}}}{{\beta }_{j}}}=\Omega {{o}^{\chi p}}\beta $            (8)

Vector β can be obtained by taking the supervision video image at the starting time point. Figure 4 shows the principle of reconstructing the coefficient matrix. Such a process effectively reduces the computational process of {ῶ¹_i,1}^t_i=1=ρx. Since the eigenvector matrix involved in the calculations in this paper is not a square matrix, β can be calculated by the pseudo-inverse procedure shown in the following equation:

$\beta ={{\Omega }^{+}}\left\{ \tilde{\gamma }_{i,1}^{1} \right\}_{i=1}^{t}$           (9)   

4.png

Figure 4. Principle of reconstructing coefficient matrix

The key operation for separating IoT supervision video images into foreground and background is to threshold the images in low frequency mode based on the feature values of the image foreground and background. Basically, the image block representing the background in the IoT supervision video image is constant in the video stream; and when tÎ{1,2,..., s}, it satisfies the equation |θ_t|≈0. In particular, it is important to note that background structure features near the spatial origin in real surveillance scenes are characterised through a single modality. |θ_t | denotes the eigenvalues of foreground structures far from the spatial origin.

Assume that the background part of the image is represented by ρ_to^θ_t^px_t, the foreground part is represented by Σ_j=Tρ_jo^θ_j^px_j, and the reconstructed coefficient matrix is represented by Y^~={ῶ¹_i,1,ῶ¹_i,2,...,ῶ¹_i,P}^t_i=1, and p={0,1,2,... ,P-1} is the time index to the (P-1) image frame. Thus, the IoT supervision video image that is divided into background and foreground structures can be represented as:

$Y\approx {{\rho }_{t}}{{o}^{\theta _{t}^{p}}}{{\beta }_{t}}+\sum\nolimits_{j=t}{{{\rho }_{j}}}{{o}^{\theta _{j}^{p}}}{{\beta }_{j}}$            (10)

The initial amplitude value x_j=ρ_j⁺{ῶ¹_i,1}^t_i=1 of the stationary background is constant and the initial amplitude of the changing foreground structure is represented by x_j=ρ_j⁺{ῶ¹_i,1}^t_i=1, ∀j≠t. Then the fully-spreading approximate IoT supervision video image sequence B^* can go through dictionary reconstruction based on the following equation:

$\left\{ {{b}_{i,j}}^{*} \right\}_{i=1,j=1}^{T,P}=C\left\{ {{{\tilde{\omega }}}_{i,j}} \right\}_{i=1,j=1}^{T,P}$               (11)

Figure 5 shows the different eigenvalues corresponding to the foreground and background parts present in the IoT supervision video image. The background position in a static state corresponds to the vicinity of the origin with a feature value of 0. The feature values of other dynamic points and moving targets correspond to positions far from the origin.

5.png

Figure 5. Scenario of different eigenvalues corresponding to foreground and background

6.png

Figure 6. Variation curve of reconstruction error for different number of dictionary elements

The approximation B₁ and B₂ of the spatiotemporal information of the IoT supervision image sequences depends on the estimation of the coefficient matrices Y₁ and Y₂. To achieve these objectives, this paper uses the L1-parametric regularization method and a fast iterative shrinkage threshold algorithm to solve equations 5 and 6. The selection of μ₁ and μ₂ determines the number of non-zero coefficients in the sparse matrix. In the dynamic modal decomposition method used in this paper, the above parameters are all set manually to obtain the desired approximation of the signal. Figure 6 shows the variation curves of reconstruction error for different numbers of dictionary elements. To better approximate the input image sequence, more reasonable µ₁ and µ₂ are set to perform noise filtering on the IoT supervision image sequence based on a dictionary containing fewer elements.

Table 1. Automatic recognition results for different sample sets

Table 1 shows the results of the automatic recognition of different sample sets. As can be seen from the results, when the detection target is static for a long time, it is difficult to detect any changes that occur in it in the future period, which reduces the F1 value of the sample’s automatic recognition. However, video images with minimal background changes have a high F1 value. The optimised dynamic modal decomposition method used in this paper has desirable F1 values for sample sets (2), (4), (7) and (9), as these sample sets are from different surveillance locations but are almost static throughout the surveillance period. The optimised dynamic modal decomposition method used in this paper can detect volumetrically different target objects while obtaining more desirable F1 values.

Table 2 gives the target recognition speed statistics for continuous images. It can be seen that the algorithm of this paper used for automatic recognition in continuous IoT supervision video images takes less than 40ms, and the performance is more satisfactory. In the training process of the proposed algorithm, the spatiotemporal decomposition of the image samples of the sample set has been carried out to complete the dimensionality reduction of the image data, which in turn effectively reduces the time of automatic image recognition and increases the efficiency of automatic recognition by more than one fifth.

To visually and clearly verify the recognition effectiveness of the algorithm proposed in this paper on the IoT supervision video images, the recognition results of four different products in the industrial IoT pipeline were compared in this paper, with the outputs corresponding to 1, 2, 3 and 4 in turn. The actual results were compared as shown by Figure 7, which reveals that 45 out of 50 product 1 samples were identified, delivering a recognition rate of 90%. Product 1 identified 44 with a recognition rate of 88%; product 3 identified 47 with a recognition rate of 94%; and product 4 identified 49 with a recognition rate of 98%, making the recognition accuracy rate quite desirable.

Table 2. Statistics of recognition speed for continuous images

7.png

Figure 7. Results of the automatic recognition and classification of different products