Visual Image Recognition of Basketball Turning and Dribbling Based on Feature Extraction

Event analysis in the process of sports aims to identify the start and end points of specific events in sports videos. Figure 1 gives an example of a complete image frame series of a basketball turning and dribbling event. The recognition of the athlete and each part of the action in the image frames lay the basis for the detection of basketball turning and dribbling.

1.png

Figure 1. Detection of basketball turning and dribbling in sports videos

Spatial and temporal information are very important for the recognition of basketball turning and dribbling of athletes. Therefore, it is necessary to select appropriate sports video image frames to extract motion features. Otherwise, it would be impossible to extract the valuable spatiotemporal features of basketball turning and dribbling. To obtain the time information of sports video, this paper introduces the optical flow image, establishes the relationship between the velocity field of basketball turning and dribbling and the gray level of the image frames, and effectively illustrates the time variation of pixels.

For an image frame in a sports video, a pixel is denoted by (a,b), the brightness of the pixel at time e+∆e is denoted by LI(a+∆a,b+∆b,e+∆e), the variation of (a,b) on axes a and b is ∆a and ∆b, respectively; e is the time interval of pixel changes; v and u are the displacement components in the horizontal and vertical directions of the optical flow of (a, b). Then, we have:

$v=\frac{d a}{d e}$      (1)

$u=\frac{d b}{d e}$    (2)

Suppose pixel (a, b) does not have brightness change as ∆e approaches 0. Then, we have:

$L I(a, b, e)=L I(a+\Delta a, b+\Delta b, e+\Delta e)$         (3)

Suppose pixel (a,b) has brightness change when ∆e is nonzero. Then, we have:

$L I(a+\Delta a, b+\Delta b, e+\Delta e)=L I(a, b, e)+\frac{\partial L I}{\partial e} \Delta a+\frac{\partial L I}{\partial b} \Delta b+\frac{\partial L I}{\partial e} \Delta e+\rho$                   (4)

Ignoring the second-order infinitesimal term in formula (4), it is possible to obtain the velocity and direction change ∇LIhq of each pixel when Δe approaches 0 at a certain moment:

$-\frac{\partial L I}{\partial e}=\frac{\partial L I}{\partial a} \frac{d a}{d e}+\frac{\partial L I}{\partial b} \frac{d b}{d e}=\nabla \operatorname{LIh} q$        (5)

Based on the above processing procedure, this paper obtains the optical flow image corresponding to the RGB image of the video frame for the player's basketball turning and dribbling.

In the recognition of athlete’s basketball turning and dribbling, it is wrong to emphasize spatial features over spatiotemporal features. The full utilization of the spatiotemporal correlation of continuous action features helps to improve the feature utilization by the action recognition model, making it easier to identify actions with inconsistent timing. Therefore, this paper strengthens the feature extraction for the recognition of basketball turning and dribbling in the same scene, improves the existing convolutional neural network, and conducts multi-feature learning based on the multi-feature learning of motion excitation and temporal aggregation of actions. Figure 4 shows the framework of the optimized model for turning and dribbling recognition.

4.png

Figure 4. Framework of the optimized recognition model

The proposed action recognition model contains a multiscale motion excitation module, and supports the following functions: the modeling of fixed actions, expansion from pixel level to feature level, and learning of spatiotemporal features.

The shape of the spatiotemporal features input to the model is represented by M, E, D, F, and Q. The size of batch processing BS is denoted by M. The time dimension and feature channel are denoted by E and D, respectively. F and Q stands for the length and width of the corresponding space shape, respectively. In addition, * and s stand for the convolutional operation and reduction scale. Then, the feature A^s after the channel size reduction satisfies:

$A^s=C O_{R E} * A, A^s \in R^{M \times E \times D / s \times F \times Q}$         (10)

This paper defines the difference between two adjacent frames A^s(e) and A^s(e−1) of the sports video image in the time dimension E as the feature-level motion N(e−1). Let CO_TR be a two-dimensional convolution for channel changes. Then, we have:

$N(e-1)=C O_{T R} * A^s(e)-A^s(e-1)$

$2 \leq e \leq E, N(e-1) \in R^{M \times D / s \times F \times Q}$           (11)

Suppose N(E)=0 at the end of the time dimension. By traversing the time dimension, the motion matrix N^e-1 of the player's basketball turning and dribbling can be constructed based on [N(1),...,N(E)]. The size of the matrix is [M,E,D/s,F,Q].

To better illustrate the feature-level motions of the athlete’s turning and dribbling, this paper extends the adjacent image frames A^s(e) and A^s(e−1) of the motion video to the three adjacent image frames A^s(e+1), A^s(e), and A^s(e−1). Similarly, A^s(e), which has completed one channel transform, is subtracted from A^s(e+1), which has completed two channel transforms, producing the difference between the adjacent image frames A^s(e) and A^s(e−1) in dimension E. That is, the feature-level motion description N(e) of the athlete’s turning and dribbling at time e can be obtained by:

$N(e)=C O_{T A} *\left[C O_{T R} * A^s(e+1)\right]$

$-C O_{T R} * A^s(e), 1 \leq e \leq E-1$

$N(e) \in R^{M \times D / s \times F \times Q}$         (12)

Similarly, we can obtain the motion matrix N^e of the size [M,E,D/s,F,Q]. Further, N^e-1 is superimposed on N^e to obtain the feature-level motion description N between the three adjacent image frames A^s(e+1), A^s(e), and A^s(e−1). To excite the motion sensitive channel of the said action, this paper summarizes the spatial information of all image frames, using the global average pooling layer:

$N^r=\operatorname{Pool}(N), N^r \in R^{M \times E \times D / s \times 1 \times 1}$           (13)

Then, a 1×1 two-dimensional convolution is adopted to restore the number of channels N*, which are about to complete pooling, to [M,E,D,1,1]. After the activation of sigmoid function, the weight X of the dynamic features of the athlete’s basketball turning and dribbling is obtained. Let ξ be the sigmoid function; CO_EX be the convolutional operation that restores the number of channels. Then, we have:

$X=2 \xi\left(C O_{E X} * N^r\right)-1, X \in R^{M \times E \times D \times 1 \times 1}$              (14)

The feature activation of the turning and dribbling features is completed by multiplying the action with X. Meanwhile, the static features of the action, which plays a role in motion recognition, are preserved based on the residual connection. Let A^p be the output of the multiscale motion excitation module; ⊗ be the channel-based multiplication. Then, we have:

$A^p=A+A \otimes X, A^p \in R^{M \times E \times D \times F \times Q}$              (15)

The basketball player turns and dribbles rather quickly. It is very difficult to establish an effective correspondence between temporal and spatial features. The problem cannot be solved by setting many local temporal convolution structures in the recognition model. Therefore, this paper constructs a squeeze-and-excitation time aggregation module, i.e., builds up a sub-convolution hierarchy. The transmission of valuable action features is highlighted by squeezing and exciting the shared features between convolutional layers. In this way, the receptive field of the time dimension is expanded, while the transfer and sharing of invalid action features is suppressed between convolutional layers.

The given features of the player’s basketball turning and dribbling are divided along the channel dimension to form 4 image frame subseries of the shape [M, E, D/4, F, Q]. The last 3 image feature subseries are processed based on temporal and spatial sub-convolutions. Suppose the A^p_i is the output of the i-th image feature subseries; CO_ZJ is the spatial sub-convolution; CO_SJ is the temporal sub-convolution; G_rt be the squeeze-and-excitation method. Then, we have:

$\begin{array}{rlrl}A_i^p & =A_i, & i=1 \\ A_i^p & =C O_{\mathrm{ZJ}} *\left(C O_{S J} * A_i\right), & i=2 \\ A_i^p & =C O_{\mathrm{ZJ}} *\left[C O_{S J} *\left(A_i+G_{r t}\left(A_{i-1}^p\right)\right)\right], i=3,4\end{array}$               (16)

After the spatial and temporal sub-convolutions, the three image frame sub-series are parallel convolution structures. To expand the receptive field of the time dimension, it is necessary to short-circuit them to form a hierarchical convolution structure. The squeeze-and-excitation operation Grt on the subseries is arranged before the residual connection is passed into the next subseries. Specifically, squeezing compresses the spatial features of each subseries into [M, E, D/4,1,1]. Then, adaptive learning is performed on the features based on two fully connected layers, producing new feature maps.

Let R be the feature map containing the importance information; G_ta be the weighted excitation function; V be the feature map passing through two fully-connected layers; Q₁ and Q₂ be the weights of the first and second fully connected layers, respectively; ε and ξ be sigmoid and ReLU activation functions, respectively. Both Q₁ and Q₂ are collectively referred to as activation function. Then, the above process can be characterized by:

$R=G_{t a}(V, Q)=\varepsilon\left(Q_2 \xi\left(Q_1 V\right)\right)$         (17)

The resulting new feature map is of the size [M,E,D/4,1,1]. In the squeeze-and-excitation time aggregation module, there are differences in the receptive fields of the subseries obtained via division. Before outputting the final results, this paper cascades the multiple outputs:

$A^p=\left[A_1^p ; A_2^p ; A_3^p ; A_4^p\right], A^p \in R^{M \times E \times D \times F \times Q}$              (18)

With the increase of the receptive fields of the image frames, the model output will contain effective spatiotemporal feature correlations. This optimization strategy improves the recognition effect of the model.