Abnormal Behavior Recognition in Classroom Pose Estimation of College Students Based on Spatiotemporal Representation Learning

In recent years, computer vision has developed rapidly, under the driving of data, computing power and algorithms. As the main objects of computer vision, image recognition and video recognition are similar in many aspects. However, far more scholars have explored videos than images. The video research not only involves detection and recognition [1], style conversion, and retrieval, but also covers target detection and tracking, personal behavior recognition, action prediction, and determining the nature of behaviors. Besides saving manpower, detecting student poses in surveillance videos [2] can alert students to abnormal poses, help them to cultivate a good learning habit, and prevent hunchback and myopia caused by improper poses, heralding the realization of unattended classroom.

Many conventional methods can identify student behaviors in class. The most common methods of behavior recognition include feature extraction, feature processing, and feature classification [3]. In the HMDB51 dataset, the improved dense trajectory (IDT) algorithm [4] has the highest recognition rate, reaching 57.2%. Xiao et al. [5] and Lu [6] were the first to apply deep learning extensively in video recognition [7]. Comparing the influence of four single-flow neural network structures on behavior recognition, it is found that a single frame of video contains lots of behavior features. Simonyan and Zisserman [8] divided the monitoring mode into a time flow and a space flow, regarded optical flow as the input of time flow, and conducted behavior recognition with dual-flow convolutional neural network (CNN), revealing that the time flow has better input performance than the original frame input. Many deep learning studies on student behavior recognition [9, 10] are based on single-flow or dual-flow neural network. In traditional empirical research, the necessary pre-test hypotheses ignore some potentially useful functions. Fortunately, the emerging technology of feature engineering helps to discover some subtle influencing factors. The intrinsic development logic of these factors can associate student trajectories outside the classroom, and greatly promote the monitoring of student behaviors.

In video teaching, both time and space information should be considered to recognize the classroom behaviors of students. How to balance the relationship between the two dimensions is the key to identifying good behaviors in time and space. Our work is unfolded based on the key information needed to recognize human behaviors in videos, that is, the edge trajectories of human body. Firstly, feature engineering was implemented to generate a series of features to describe the lifestyles, income patterns, and Internet use patterns of students. Then, the features of different student groups were identified through K-means clustering (KMC) [11]. On classroom performance, the edge features of video frames were taken as the basis for the spatial dimension of behavior recognition, and the three-dimensional CNN (C3D) was adopted to fuse the spatiotemporal information of continuous frames. On this basis, repeated experiments were conducted, and a robust abnormal behavior recognition algorithm was designed to timely detect and prewarn abnormal student poses.

3.1 Problem definition

When a college student makes a payment with his/her campus card, his/her personal information is recorded. If he/she uses his/her identity to access the Internet or rent a house, his/her personal information will be added to some data records. These records will continue to increase with the elapse of time. The recorded data can accurately reflect the daily behaviors of the college student.

The scores of college students are closely correlated with self-discipline and time management. Good students generally have different behavior patterns from average students [20]. Therefore, the scores of college students can be reflected in daily life activities, such as smart card usage, Internet access, and cellphone/computer interactions.

3.2 Preparatory work

The functional areas are distributed in different places on campus. For example, the dormitories might scatter across the college, and have different names. Let S_γ be the campus of the target college; F_η be the functional areas on campus, including dormitories, canteens, classrooms, libraries, yards, bathrooms, etc.; (p.lo,p.fu) be the λ-th position p(γ, λ) on campus, where p.lo is the position of physical position p, p.fu is the functional area of p; c(·) be the function to transform p.lo into functional area p.fu. The time and position information are primarily extracted from consumption data, network service data, and access control data.

The activity of a student A can be labeled by his/her time tag t_A, functional area pÎF_η, activity area bÎB_φ, as well as other relevant attributes. The activity sequence of A can be vectorized as b. In addition, the consumption behavior consists of such attributes as consumption amount, and card balance. A charging behavior at least covers information like “maximum amount”. After being extracted from the consumption data, network service data, and access control data, the time and position of each student in every activity can be sorted in that period, without changing the spatiotemporal sequence of students.

The activity sequence of student A contains both spatial order and time order. Here, the activity sequence is defined as Aseq={(t₁, q₁),…,(t_k, q_k)}, where t_i<t_j (i<j) is time, q_iÎq_ϕ is the i-th item in the sequence, and q_i could be equal to q_j.

For the given activity sequence Aseq={(t₁,p₁),…,(t_k, p_k)} and parameter ξ, if (t_i, P_i) and (t_i+1,P_i+1) satisfy P_i=P_i+1 and if |t_i-t_i+1|<ξ, then (t_i, p_i) and (t_i+1, P_i+1) belong to the same position.

The trajectory Tra of student A is a subset of Aseq: TraÎAseq.

Using the node information in behavioral data, the original data were converted to semantic trajectories. Following the time-based trajectory segmentation method, this paper correctly extracts the trajectory from the activities of each student on a day [21-23]. Note that the day refers to the “day” for the student to complete a set of activities, rather than a natural day (from 00:00 to 00:00). For example, the “daily” trajectories of a student could cover {(10pm, November 3, classroom)-(11pm, November 3, bedroom)…(0am, November 4, bedroom)}. Although the trajectories cover two days, all the activities are clearly completed within the same day. Thus, the daily journey of each student can be defined as follows.

Here, the behavior trajectory is defined as Tra={(t₁, q₁),…,(t_k, q_k)}, and the daily behavior trajectories as DTra={(t₁,q₁),…,(t_s,q_s)}. The daily trajectories are organized into a trajectory set (DTraÎTra). In the Tra, there must exist a unique (t_j, q_j) such that (t₁, q₁)=(t_j, q_j), and "_nÎ{1,2,…,S-1},(t_1+n,q_1+n)=(t_j+n, q_j+n).

The same activity recorded by the campus information system may contain multiple data that needs to be checked. The trajectory of such an activity can be expressed as $\operatorname{Tr} a P=q_{1}^{\prime} \stackrel{\Delta t_{1}^{\prime}}{\longrightarrow} q_{2}^{\prime} \stackrel{\Delta t_{2}^{\prime}}{\longrightarrow} \cdots \stackrel{\Delta t_{u-1}^{\prime}}{\longrightarrow} q_{u}^{\prime}$, where $q_{i}^{\prime} \in\left\{q_{j}\right\}$. If (t_j, q_j) corresponds to (t_i, q_i), and $q_{i+1}^{\prime}$  corresponds to (t_j, q_j), then $\Delta t_{i}^{\prime}=t_{j}-t_{i}$.

3.3 Cluster analysis

For the given ∂={∂₁, ∂₂,…, ∂_N} and β={β₁, β₂,…, β_K}, if ∂ is the n-th instance of ∂_n, then β_K is the k-th cluster in β. Then, ∂_n={∂_n1, ∂_n2,…, ∂_nM}, where ∂ nm is the m-th eigenvalue. Moreover, β₁Èβ₂È…Èβ_K=∂, and β₁Çβ₂Ç…Çβ_K=ϕ.

For the given ∂={∂₁,∂₂,…,∂_IV} and β={β₁, β₂,…, β_k}, if S_∂ is the n-th ∂, then β_k is the k-th cluster in β. β_k={β_k1, β_k2,…, β_kM}, k=1,2,...,K.

The K-means algorithm aims to maximize the intra-cluster similarity and minimize inter-cluster similarity. Therefore, the constraint function is the sum of the distance between the intra-cluster instances and the cluster head:

$P(\mathbf{U},\beta )=\sum\limits_{k=1}^{K}{\sum\limits_{n=1}^{N}{{{u}_{nk}}}}\sum\limits_{m=1}^{M}{d}\left( {{x}_{n~\text{m}}},{{\beta }_{k~\text{m}}} \right)$     (1)

where, U is an N´K matrix; u_nkÎ{0,1} (If u_nk=1, then the n-th instance belongs to the k-th cluster); $\sum_{k=1}^{K} u_{n k}=1, n=1,2, \cdots, N ; \beta$ consists of K clusters, i.e., β={β₁,β₂,…,β_k}; d(x_nm,β_km) is the distance between an intra-cluster instance and cluster head. The constraint function can be rewritten by Euclidean distance:

$P(\mathbf{U},\beta )=\sum\limits_{k=1}^{K}{\sum\limits_{n=1}^{N}{{{u}_{nk}}}}\sum\limits_{m=1}^{M}{{{\left( {{x}_{n~\text{m}}}-{{\beta }_{k~\text{m}}} \right)}^{2}}}$     (2)

Obviously, each function is equal in formula (2). In fact, only some special functions are useful. Thus, functional selection was adopted to remove redundant, noisy functions [24]. In addition, feature weighting is a generalization technology, which assigns a weight to each feature within the range of [0, 1] rather than delete that feature [25]. Therefore, some scholars held that, before applying the K-means algorithm, each feature should be given a weighted tag, and the weights of M features should be defined as W=(ω₁, ω₂,…, ω_M) Then, formula (2) can be rewritten as:

$P(\mathbf{U},\beta )=\sum\limits_{k=1}^{K}{\sum\limits_{n=1}^{N}{{{u}_{nk}}}}\sum\limits_{m=1}^{M}{w_{m}^{\beta }}{{\left( {{x}_{n~\text{m}}}-{{\beta }_{k~\text{m}}} \right)}^{2}}$      (3)

where, WÎ[0,1]; $\sum_{m=1}^{M} \omega_{m}=1$; β is a user defined parameter. Under the constraint that the sum of all weights equals 1, the feasible approach is to express the minimum value of formula (3) as:

${{w}_{m}}=\frac{1}{\sum\limits_{t\in F}{{{\left[ {{D}_{m}}/{{D}_{t}} \right]}^{1/(\beta -1)}}}}$     (4)

where, $D_{m}=\sum_{k=1}^{K} \sum_{n=1}^{N} u_{n k}\left(x_{n m}-c_{k m}\right)^{2}$ is the sum of variances of each cluster.

By modifying the basic K-means algorithm, this paper solves the weight difference between features. Drawing on subjective weight V and objective weight W, an integrated weight can be defined as γ=(γ₁, γ₂,…, γ_M) with $\gamma_{m}=\frac{w_{m} v_{m}}{\sum_{m=1}^{M} w_{m} v_{m}}$. Whereas A=(a_l, a₂,⋯, a_n) and B=(b_l, b₂,⋯, b_m), n≤m, if there exists a sequence satisfying 1≤j₁≤j₂≤···≤j_n≤m, a₁Îb_j1, a₂Îb_j2 then, A is the subsequence of B, and B is the hyper sequence of Yu [26]. The subsequences satisfy the following assumptions: (1) the items in every subsequence exist in the hyper sequence; (2) the items in every subsequence follow the same order as they are in the hyper sequence. For example, S=<a(abc)(ac)d(cf)>, <a(abc)> is a subsequence of S. For sequences A=(a_l,a₂,⋯,a_n) and B=(b_l,b₂,⋯,b_m), n≤m, if a_l=b_l, a₂=b₂,…, a_n-1=b_n-1 and a_nÍb_n, then A is the prefix of B.

5.1 Experimental environment

The experimental environment is as follows: the central processing unit (CPU) is Intel hexacore i7-8750h; the graphics card is Gigabyte GeForce RTX 3080 EAGLE OC 10G; the memory is 16G; the operating system is Ubuntu 18.04; the programming language is Python; the deep learning framework is TensorFlow 2.0 (GPU version).

5.2 Dataset

Our dataset comes from the opensource code dataset of Sun et al. [34]. It covers the student behaviors identified, detected, and labeled in class. In total, there are videos on 11 subjects taught in 128 classes. The dataset was split into a detection dataset, two recognition datasets, and a title dataset. The detection dataset contains 4,542 samples for time detection and 3,343 samples for action detection; the two recognition datasets respectively contain 4,276 samples and 4,296 samples. The datasets are representative and authentic, because all the actions in them are spontaneous actions by students in actual classrooms. The datasets were uploaded to https://github.com/BNU-Wu/Student-Class-Behavior-Dataset.

5.3 Training results analysis

(1) Video segmentation

Each video was divided into several frames. Following the edge extraction method in Section 2, the single-pixel, 3-pixel, and 5-pixel edges were extracted from the video frames. Then, the data were split into a training set and a test set by the ratio of 7:3. Our model was trained on the training set and evaluated on the test set.

(2) Model training

In each iteration, 8 samples were randomly selected, and 16 continuous frames were extracted by random from each sample. Then, the spatiotemporal data acquired by KMC from the students were fused by the scheme of Lu [6], Wu et al. [13]. To evaluate the recognition effect, each model was trained and evaluated by the training and test sets. The improved C3D model was compared with the original C3D in terms of accuracy. Figure 3 compares the two models with 3-pixel edges as the input. It can be seen that the improved C3D is much more accurate than the original model; through the improvement, the number of parameters was reduced by half, and the training speed and reasoning speed were almost doubled.

(3) Model comparison with different inputs

Next, the generalization ability of our model was compared with four different inputs: original video frames (Figure 4(a)), single-pixel edges (Figure 4(b)), 3-pixel edges (Figure 4(c)), and 5-pixel edges (Figure 4(d)). It is generally believed that the model generalization ability is positively correlated with the precision of the training set and the size of the test set.

As shown in Figure 4, our model reached an accuracy as high as 95% on the training set, with the inputs of original video frames and single-pixel edges, but its accuracy on the test set with these inputs was 10% lower. Therefore, the two types of inputs are not good for the generalization ability of our model. When 3-pixel edges were inputted, the accuracy on test set gradually approximated that on training set, with the growing training steps, after 2,400 iterations, the accuracy on test set rose to around 93%.

Finally, the model accuracies on four different inputs were compared on the test set. It can be seen from Figure 5 that, after 800 training steps, the accuracy with the input of 3-pixel edges was much higher than that with another input; after 2,400 training steps, the accuracy with the input of 3-pixel edges increased to 93%, way higher than that (85%) with other types of inputs. Hence, 3-pixl edges are the best input features for video behavior recognition. In Figure 5, the four curves had basically the same trend. The behavior recognition rates with edges as inputs were comparable and even better than the rate with original video frames as inputs. As a result, our assumption is verified that the key information of human behaviors in videos is contained on the body contour.

3.png

Figure 3. Accuracies of original and improved C3Ds

4a.png

4b.png

Figure 4. Comparison between accuracies on training set and test set

5.png

Figure 5. Test set accuracies with four different types of inputs

5.4 Recognition effect

Despite limitations on some behavior features, some behaviors are visually similar, which poses a challenge to pose identification and detection. Our method could distinguish between two different poses, in spite of their visual similarity, whether in singe person recognition or multi-person recognition.

As shown in the upper subgraph of Figure 6, our method accurately judged whether the student is learning or using cellphone, with an accuracy of 0.96. This is because our method incorporates external spatiotemporal representation data into the 3D convolution.

As shown in the lower subgraph of Figure 6, although multiple students sat in different places in the classroom and their images were of different sizes in the video, our method could accurately recognize their pzoses, for our model has a certain prior knowledge through the KMC of their common behaviors on campus.

6.png

Figure 6. Recognition effect