Student Behavior Identification During Practice and Training Based on Video Image

After the impact of COVID-19 epidemic, the higher vocational education in China has posed some new requirements for talent cultivation, and an objective of “setting curriculum under ideological and political guidance” has been proposed for the purpose of training students to become virtuous and skilled talents with the ability to cope with the epidemic, to serve our people, and be good at management and operation, so that in the future, they could grow into honourable laborers who can care for people, have new ideas of management and operations, and fight the epidemic with courage. Practice and training is a major means of labor education, in order to create new content and approach for labor education in higher vocational colleges, labor education must be pragmatic and engaging. The content setting of students’ practice and training is very important, so this paper took the property management major as the subject and divided the content of practice and training into three aspects:

The first content is owner management. One basic task of owner management is the management of property owners, specific content of this task includes: managing owner information such as the building unit number, room area, house orientation, the name, birthday, personality, contact number, and work unit of property owners and their family members. All these information should be registered and documented properly for easy inquiry in the future.

The second content is security management. The focus of security management is to prevent the occurrence of various accidents. Modern security management combines artificial prevention measures with technical prevention measures, wherein artificial prevention is carried out by personnel of special job posts, for example, typical security-related posts of a building include the VIP security posts, lobby security posts, mobile posts, parking lot security posts, and fire prevention posts; technical prevention adopts modern technologies to support artificial measures and carry out security management tasks, examples include the monitoring system, smoke sensor, displacement sensor, smoke sensor, temperature sensor, and infrared sensor, etc.

The third content is equipment maintenance and maintenance management. The repair and maintenance of equipment is an important task starting from the acceptance check of the equipment. Once an equipment is taken over by the property management department, its warranty scope and time should be figured out, and the maintenance work of the equipment should be checked clearly. At the same time, equipment operation procedures and equipment failure emergency response procedures should be formulated; monthly, quarterly, and annual maintenance plans should be made; equipment management system and equipment room management system should be established; a machine account should be set for each equipment, the parameters of each equipment, as well as each maintenance, should be properly recorded, so that every operator could have a clear understanding of the equipment at a glance. Moreover, a label should be made for each equipment and all equipment should be numbered, in this way, the records could be accurate enough. One last thing, the energy consumption budget should be made before the running of the equipment to achieve economic operation.

Research content of this paper is to help teachers know about students’ status of practice and training by identifying their actions and intentions during practice and training activities. Taking the property management major as an example, this paper can assist teachers to check whether the students have completed tasks of owner management, security management, and equipment maintenance management in various practice and training scenarios. In our study, a task of asking students to cooperate with each other to deal with an equipment failure emergency was set to simulate a scenario of student practice and training, during which the actions between students when they pass equipment maintenance tools and components to each other were captured. During this task, students were asked to work in pairs to complete jobs such as deliver and fetch the equipment maintenance tools and components, before the task, they were required to watch a video showing the specific steps, then they were asked to repeat the equipment failure emergency response task for multiple times until they reached the required practice and training level of professional property management.

In the research of intention identification of student behavior during practice and training, only identifying static images or only considering the spatial features of actions can result in loss of temporal features of the actions, for instance, when a student tightens or loosens a screw, if we only observe the behavior in a single frame, we can not judge whether he/she is tightening the screw or loosening it, and whether the behavior is an effective action of equipment failure emergency response or not is pending for judgement. Therefore, behavior identification based on video image plays a very important role in identifying student behavior during practice and training. In our study, the dynamic convolution kernel was optimized based on the collected video image data, and a lightweight convolution network model was built for behavior identification by terminal devices with less parameters used in specific practical scenarios.

image010.png

Figure 2. Calculation flow of 3D dynamic convolution

Conventional 3D dynamic convolution can superimpose all channel dimensions and time dimensions of the collected video images to form one dimension for processing, however, inevitably, this will introduce some redundant information that can affect behavior identification, so this paper proposed a kind of optimized 3D dynamic convolution kernel based on efficient attention to solve this problem, and its calculation flow is given in Figure 2.

Assuming: the size of input video is D_im×F×Q×P, by superimposing time dimension and channel dimension, the input feature of video images can be superimposed into D_im×P×F×Q, after that, the superimposed dimension can be compressed at the scale of γD_im×P×F×Q, wherein the value range of β is [0,1]. In order to reduce redundant information and retain useful information to the great extent, in this paper, down-sampling processing was performed on the spatial feature of video images after subjected to dimension compression at the scale of γD_imP×μF×μQ, the processed video image features were then superimposed at the scale of γD_imP×μ²FQ. Furthermore, the output result was processed by the fully-connected layer and the adaptive pooling layer, and finally the dimension of video image features had been reduced to L, namely the number of dynamic convolution kernels, at last, the attention weight was calculated by the Softmax function.

Assuming: U(.) represents the superimposing operation of channel dimension and time dimension, R(.) represents the dimension compression process of input features, H(.) represents the process of adaptive average pooling which is used for downsampling the feature vector; G₁ and G₂ represent two layers of fully-connection operations in the module, ε(.) represents the ReLu nonlinear function, then, for an input video of students’ practice and training C∈R^D_im^*F*Q*P, its dynamic weight can be calculated based on the following formula:

$\beta =Soft\max \left( {{G}_{2}}\varepsilon \left( {{G}_{1}}\left( H\left( R\left( U\left( C \right) \right) \right) \right) \right) \right)$    (7)

The random channel-time dimension was adopted to perform redundancy-removal dimension compression on 3D dynamic kernels to reduce the superimposed time and channel dimensions, specifically, γD_imP dimensions were randomly picked from D_imP dimensions of video images:

$\overset{\scriptscriptstyle\frown}{E}=R\left( U\left( C \right) \right)$    (8)

$\overset{\scriptscriptstyle\frown}{C}=\left\{ {{{\overset{\scriptscriptstyle\frown}{C}}}_{0}},{{{\overset{\scriptscriptstyle\frown}{C}}}_{1}},...,{{{\overset{\scriptscriptstyle\frown}{C}}}_{\gamma {{D}_{im}}P}} \right\}$    (9)

$\overset{\scriptscriptstyle\frown}{C}\subseteq {{C}^{{{D}_{im}}P}}$    (10)

image015.png

Figure 3. Flow of random dimension compression

Although the above method can significantly reduce the redundant information of video images in terms of time and channel, an improper compression scale can easily cause information loss of student behavior during practice and training in the time dimension, so this paper directly performed random dimension compression on the channel dimension to solve this problem, and the flow of this random dimension compression is shown in Figure 3. Assuming: Ĉ represents the random dimension compression operation, then the formulas for calculating dynamic weight after compression strategy optimization are given below:

$\beta =Soft\max \left( {{G}_{2}}\varepsilon \left( {{G}_{1}}\left( H\left( U\left( R\left( C \right) \right) \right) \right) \right) \right)$    (11)

$\overset{\scriptscriptstyle\frown}{C}=R\left( C \right)$    (12)

$\overset{\scriptscriptstyle\frown}{C}=\left\{ {{{\overset{\scriptscriptstyle\frown}{C}}}_{0}},{{{\overset{\scriptscriptstyle\frown}{C}}}_{1}},...,{{{\overset{\scriptscriptstyle\frown}{C}}}_{\gamma {{D}_{im}}}} \right\}$    (13)

$\overset{\scriptscriptstyle\frown}{C}\subseteq {{C}^{{{D}_{im}}}}$    (14)

Therefore, in case that bias is not considered and output dimension is Ḋ_out, the parameter size of the fully connected layer is μ²γD_imPFQḊ_out, which can be turned into μ²γD_imPḊ_out by adopting 2D convolution with a l-sized kernel, such operation can reduce parameter size and make the fully connected layer lighter, the following formula calculates the dynamic attention weight after improvement:

$\beta =Soft\max \left( {{G}_{2}}\varepsilon \left( {{G}_{1}}\left( H\left( R\left( U\left( C \right) \right) \right) \right) \right) \right)$    (15)

If the adaptive convolution operation of convolution kernel size is adopted, then the convolution kernel size at this time can be calculated by the following formula:

$l=\phi \left( {{{\dot{D}}}_{im}} \right)={{\left| \frac{{{\log }_{2}}\left( {{{\dot{D}}}_{im}} \right)}{\alpha }+\frac{y}{\alpha } \right|}_{odd}}$    (16)

When l value is solved in the attention mechanism, the Ḋ_im of l value in G₁(.) and G₂(.) is γḊ_imP.

The improved dynamic convolution kernel is plug-and-play and can be applied to lightweight convolution neural networks. The depth-separable convolution with a kernel size of 3×3×3 can be replaced by a dynamic convolution kernel. Assuming: β_i^* represents the weight of the i-th attention weight, L represents the number of dynamic convolution kernels, then the convolution process is given by the following formula:

$g\left(a_p\right)=\left[\begin{array}{l}\bar{B}_1^{\prime} \\ \bar{B}_2^{\prime} \\ \cdots \\ \bar{B}_{D_{\mathrm{im}}}^{\prime}\end{array}\right]=\left[\beta_1^*\left[\begin{array}{cccc}\bar{q}_{11}^1 & 0 & \cdots & 0 \\ 0 & \bar{q}_{22}^1 & \cdots & 0 \\ \ldots & \cdots & \ddots & \cdots \\ 0 & 0 & \cdots & \bar{q}_{D_{im} D_{im}}^1\end{array}\right]\right]$$+\beta_2^*\left[\begin{array}{cccc}\bar{q}_{11}^2 & 0 & \cdots & 0 \\ 0 & \bar{q}_{22}^2 & \cdots & 0 \\ \cdots & \cdots & \ddots & \ldots \\ 0 & 0 & \cdots & \bar{q}_{D_{im} D_{im}}^2\end{array}\right]$$\left.+\ldots+\beta_l^*\left[\begin{array}{cccc}\bar{q}_{11}^l & 0 & \cdots & 0 \\ 0 & \bar{q}_{22}^l & \cdots & 0 \\ \cdots & \cdots & \ddots & \cdots \\ 0 & 0 & \cdots & \bar{q}_{D_{im} D_{im}}^l\end{array}\right]\right]\left[\begin{array}{l}A_1 \\ A_2 \\ \cdots \\ A_{D_{\text {im }}}\end{array}\right]$       (17)

When 3D point convolution of structure blocks in the network is replaced by dynamic convolution kernel, the convolution process can be expressed as:

$\left.g\left(a_p\right)=\left[\begin{array}{l}\bar{B}_1^{\prime} \\ \bar{B}_2^{\prime} \\ \cdots \\ \bar{B}_{D_{\text {im }}}^{\prime}\end{array}\right]=\left[\begin{array}{cccc}\bar{q}_{11}^1 & \bar{q}_{12}^1 & \cdots & \bar{q}_{1 D_{\text {im }}}^1 \\ \bar{q}_{21}^1 & \bar{q}_{22}^1 & \cdots & \bar{q}_{2 D_{\text {im }}}^1 \\ \cdots & \cdots & \ddots & \cdots \\ \bar{q}_{D_{\text {out }} 1}^1 & 0 & \cdots & \bar{q}_{D_{\text {out }} D_{im}}^1\end{array}\right]\right]$$+\beta_2{ }^*\left[\begin{array}{cccc}\bar{q}_{11}^2 & \bar{q}_{12}^2 & \cdots & \bar{q}_{1 D_{\text {im }}}^2 \\ \bar{q}_{21}^2 & \bar{q}_{22}^2 & \cdots & \bar{q}_{2 D_{\text {im }}}^2 \\ \cdots & \cdots & \ddots & \cdots \\ \bar{q}_{D_{\text {out }} 1}^2 & \bar{q}_{D_{\text {out }} 2}^2 & \cdots & \bar{q}_{D_{\text {iout }} D_{\text {im }}}^2\end{array}\right]$$\left.+\ldots+\beta_l^*\left[\begin{array}{cccc}\bar{q}_{11}^{\mathrm{L}} & \bar{q}_{12}^{\mathrm{L}} & \cdots & \bar{q}_{1 D_{\text {im }}}^L \\ \bar{q}_{21}^{\mathrm{L}} & \bar{q}_{22}^{\mathrm{L}} & \cdots & \bar{q}_{2 D_{\text {im }}}^L \\ \ldots & \cdots & \ddots & \cdots \\ \bar{q}_{D_{\text {out }}1}^{\mathrm{L}} & \bar{q}_{D_{\text {out }} 2}^{\mathrm{L}} & \cdots & \bar{q}_{D_{\text {iout }} D_{\text {im }}}^L\end{array}\right]\right]\left[\begin{array}{l}A_1 \\ A_2 \\ \cdots \\ A_{D_{\text {im }}}\end{array}\right]$         (18)

According to above formula, compared with the convolution kernel before improvement, the improved convolution kernel only corrects the weight size when performing depth-separable convolution operations, when performing point-wise convolution, it can be considered as ordinary convolution with a kernel size of 1.

This paper took the property management major as the subject and studied the identification of student behavior during practice and training based on video image. At first, students’ practice and training content was divided into three aspects, a task of asking students to cooperate with each other to deal with an equipment failure emergency was adopted for the research, and a research idea of helping teachers figure out students’ status during practice and training via identifying their actions and intentions during the said activities was determined. Then, a few pre-processing operations were performed on the captured video images of student behavior during practice and training, including removing abnormal image frames, filtering, and aligning, etc. After that, based on the collected video image data, the dynamic convolution kernel was improved and optimized, and a lightweight convolution network model was built for identifying student behavior during practice and training. In the experiment, this paper designed specific actions for the equipment fault emergency response task, pre-processed the collected video images of 6 steps, and verified the validity of the pre-processing steps; then a 4-layer network model was built and subjected to random dropout experiment. At last, performance of the proposed model was compared with conventional CNN and LSTM in experiment, and the results proved that the identification accuracy of the proposed model for 7 actions is higher than that of CNN and LSTM.