Deep Learning-Based Standardized Evaluation and Human Pose Estimation: A Novel Approach to Motion Perception

Traditional dance teaching emphasizes on the consistency of dancing movement, ignoring the students’ individual characteristics. Whereas, personalized and diversified dance education is oriented to different objects and goals according to students’ individual conditions, educational backgrounds, and developmental needs. The traditional dance teaching model cannot effectively achieve the goal of cultivating diversified dance talents [1]. In particular, the training of dance movement, as one of the fundamental parts in dance teaching, covers a wide range of standard movements such as jumping jack, flick Jump, leg curl jump, and knee lift jump etc. But in the dance training, some incorrect poses may influence the training quality, the students’ dance scores, and even lead to delayed graduation etc. [2]. Therefore, it is necessary to identify the human pose in the training process and timely solve the problems existing in dance training in colleges, thus improving the effect of dancing training. Currently the research on the real-time identification of human pose has received wide attention from all walks of life. Especially, following the development of big data and image information processing technology, a real-time human pose estimation method for dance training has been developed to more effectively correct the human pose in college dance training, and improve the training effect. In view of the above, the study on the relevant HPE for college dance training is of great significance for optimizing the teaching quality of dance training [3]. This paper aims to explore a highly robust, high-precision method for correcting the human poses through the study on the standard pose movement of dances.

In summary, it urgently needs to solve the problem of how to express the subjective experience of professional evaluators quantitatively and reproduce it intuitively in the dance teaching and evaluation. To solve the problems above, this paper investigates a deep learning-based HPE and evaluation method. First, OpenPose is applied to achieve the recognition of human skeleton and key points, and to express the body language of the test subjects quantitatively; then, a DNN-based pose estimation method is established to obtain the information of human pose and the main characteristics of the actions being completed by the test subjects, while considering the individual differences the quantitative expression library of standard dance movements is built by collecting standard dance movements of different professional dance teachers; the evaluation robustness is obtained by learning the allowable feature differences; finally, the feature description of dance movement and the method of determining the differences were developed to realize its quantitative evaluation and intuitive feedback mechanism, thereby better monitoring the students’ learning.

Usually, there are two ways to obtain skeleton points: one is to obtain point cloud data through special devices such as depth sensors [22], and then perform pose recognition through the point cloud, to obtain the skeleton and key points, which can ensure high recognition accuracy, but relying on special equipment with a huge amount of computation, and has a depth-of-field limitation, being unfavorable to the environment of multiple dancing [23]; the other one is realized by ordinary camera such as OpenPose pose recognition algorithm, which can obtain high accuracy under various working conditions with no need of special equipment, and also gradually becomes the mainstream scheme of pose recognition.

3.1 Skeleton and key point extraction

To ensure the accuracy, this paper selects OpenPose [19] as the base recognition method for skeleton and key point extraction. OpenPose proposes the first bottom-up association score representation through Part Affinity Fields (PAFs) [24], which is a set of two-dimensional vector fields encoding the position and orientation of the limbs on the image domain. The system takes the color image in $\omega \times h$ as input and generates the two-dimensional anatomical key point locations of each person in the image as output. First, the feedforward network predicts both a set of 2D confidence maps $S$ of body part locations and a set of 2D vector fields $L$ of part affinities that encode the degree of association between parts. The set $S=$ $\left(S_1, S_2, \cdots, S_J\right)$ has a confidence map, one for each part, where $S_j \in \mathbb{R}^{\omega \times h}$. The set $L=\left(L_1, L_2, \cdots, L_C\right)$ has a vector field, one for each limb (limb), where $L_C \in \mathbb{R}^{\omega \times h \times 2}, c \in\{1, \cdots, C\}$. Finally, the confidence maps and affinity fields are parsed by greedy inference to output the two-dimensional key points of all people in the image.

3.2 Pose representation

In the pose representation, the positions of all body joints $k$ are encoded into a pose vector $y$, to be defined as $y=$ $\left(y_1^T, y_2^T, \ldots, y_i^T, \ldots, y_k^T\right)^T, i \in\{1, \ldots, k\}$, where $y_i$ contains the $x$ and $y$ coordinates of the $i^{t h}$ joint. The labeled image is denoted by $(x, y)$ where $x$ represents the image data and $y$ is the ground truth pose vector.

Since the joints of the human body are represented by the absolute coordinates in the image, it is beneficial to perform normalization in the $b$ containing the human body ROI. Generally, the ROI's $b$ can represent the complete image containing the human body, such that $b$ is defined by the center $b_c \in \mathbb{R}^2$, width $b_w$, and height $b_h: b=\left(b_c, b_w, b_h\right)$. The joints $y_i$ can then be translated by the box center and scaled by the box size, which is called normalization by $b$.

$N\left(y_i ; b\right)=\left(\begin{array}{cc}1 / b_\omega & 0 \\ 0 & 1 / b_h\end{array}\right)\left(y_i-b_c\right)$               (1)

Furthermore, the same normalization can be applied to the pose vector $N(y ; b)=\left(\ldots, N\left(y_i ; b\right)^T, \ldots\right)^T$ for a normalized pose vector. Afterwards, the cropping of the image is represented by the bounding box B using $N(x ; b)$, which is actually the image normalization by the box.

3.3 DNN regression-based pose estimation

In this study, the pose estimation problem is considered as a regression. By using the function $\psi(x ; \theta) \in \mathbb{R}^{2 k}$, the image $x$ is regressed to a normalized pose vector, where $\theta$ is the model parameter. Thus, based on the normalized transformation from Eq. (1), the pose prediction $y^*$ is read in absolute image coordinates as:

$\mathrm{y}^*=N^{-1}(\psi(N(x)) ; \theta)$                  (2)

where, $\psi$ denotes the property and complexity of the method. Such a convolutional network consists of several layers - each layer is a linear transform, followed by a nonlinear transform. Considering that DNN architectures excel in classification and localization problems, the architecture of $\psi$ is built according to the Ref. [5], as shown in Figure 1. The entire network consists of seven layers.

1.png

Figure 1. Schematic diagram of DNN-based pose regression framework

C denotes the convolutional layer, LRN (Local Response Normalization) is the local response normalization layer [25], $\mathrm{P}$ is the pooling layer, and $\mathrm{F}$ is the fully connected layer. Only the $\mathrm{C}$ and $\mathrm{F}$ layers contain learnable parameters, while the rest are parameter less. Both $\mathrm{C}$ and $\mathrm{F}$ layers consist of a linear transformation and a nonlinear transformation (using rectified linear units (PRelu)). For the C layers, the size is defined as width $\times$ height $\times$ depth, where the first two dimensions are spatially meaningful, while the depth defines the number of filters. The filter sizes are $11 \times 11$ and $5 \times 5$ for the first two layers and $3 \times 3$ for the remaining three. The pooling, when applied to all three layers, helps to improve performance despite the reduced resolution. The input to the network is a $220 \times 220$ image and stride $=4$.

In the training, the difference with the Ref. [5] can be defined as the loss (loss). To avoid the classification loss, the linear regression is trained on the last network layer to predict the pose vector by minimizing the distance $L_2$ between the predicted and the true pose vector. When the true pose vector is defined in absolute image coordinates and the pose size varies from image to image, the training set $D$ is normalized using the normalization method in Eq. (1) as follows:

$D_N=\{(N(x), N(y)) \mid(x, y) \in D\}$                (3)

Then the L₂ loss of the optimal network parameters is given as:

$\arg \min _\theta \sum_{(x, y) \in D_N} \sum_{i=1}^k\left\|y_i-\psi_i(x ; \theta)\right\|_2^2$              (4)

where, the parameter $\theta$ is optimized for the use of backpropagation in a distributed online process. The training parameters are set to be mini-batch $=128$, and adaptive gradient is updated and calculated with reference to the study [26]; learning rate $=0.0005$. due to the large number of parameters in the model and the relatively small size of the dataset, the images are enhanced by lots of random transformations (rotation, mirroring, etc.) in the training.

3.4 Cascaded pose regression

DNN regression-based pose estimation is made for the complete image, so it is context dependent. However, due to its fixed input size of 220×220, the network has limited ability to view details - it learns filters that capture the coarse pose attributes. Despite its fast and coarse estimate of pose, it is not accurate enough to eventually localize each key joint of the body. Also, this can be done by changing the size of the input image, but leading to a significant computational complexity. Therefore, it is a good solution to improve the accuracy by training a cascaded pose regressor.

First, the initial pose is estimated, and then additional DNN regressors are trained to predict joint position changes and displacement. In this way, each subsequent stage can be considered as a refinement of the current predicted pose, as shown in Figure 2:

2.png

Figure 2. Framework of cascaded pose regressors based on DNN

Thus, only the predicted subgraph of the joint position is cropped each time, and then the key regression function is applied to that subgraph. In this way, even if the size of the input image cannot be changed, we still obtain the detailed features on a finer scale, and then improve the accuracy because the subgraph contains only the joint image as much as possible.

Besides, the same network architecture is used for all stages of cascade, so it only needs to learn different network parameters. The stage $s \in\{1, \ldots, S\}$ denotes the network parameters of the total $\mathrm{S}$ cascade stage, and $\theta_s$ is the learned network parameters. Then, the positional displacement regression function reads $\psi\left(x ; \theta_s\right)$. To specify a given joint position $y_i$, we consider a joint box $b_i$ that captures the surrounding sub-images of $y_i: y_i: b_i(y ; \sigma)=$ $\left(y_i, \sigma \operatorname{diam}(y), \sigma \operatorname{diam}(y)\right)$ with the $i$-th joint as the center and $\theta$ as the scale of the pose diameter. The pose diameter $(y)$ is defined as the distance between relative joints on the human torso, such as the left shoulder and the right hip, depending on the specific posture definition and data set.

Thus, at stage $s=1$, starting from an enclosing box $b^0$, it either contains the complete image or is obtained by the human detector. Thus, an initial pose Stage 1 is obtained:

$\begin{align}  & Stage\ 1:{{y}^{1}}\leftarrow {{N}^{-1}}(\psi (N(x;{{b}^{0}});{{\theta }_{1}});{{b}^{0}}) \\  & \quad \cdots  \\  & Stage\ S:y_{i}^{s}\leftarrow y_{i}^{s-1}+{{N}^{-1}}(\psi (N(x;b);{{\theta }_{s}});b) \\  & \quad \quad \quad \quad \quad \quad for\quad b=b_{i}^{(s-1)} \\  & \quad \quad \quad \quad \quad b_{i}^{s}\leftarrow (y_{i}^{s},\sigma diam({{y}^{s}}),\sigma diam({{y}^{s}})) \\ \end{align}$               (5)

In each subsequent stage $s \geq 2$, for all joints $i \in\{1, \ldots k\}$ a regression function is applied to the sub-images defined in the previous stage $y_i^s-y_i^{(s-1)}$. First, a regression is applied towards the refined displacement $b_i^{(s-1)}$. Then, the new joint box $b_i^s$ is estimated. The network parameters $\theta_1$ are trained according to the Eq. (4). In the next stages ( $s \geq 2$ ), the training is performed in the same way, with only one big difference. In the training example $(x, y)$, each joint $i$ uses a different enclosing box ${{y}_{i}}^{(s-1)},\sigma \text{diam}(y),\sigma \text{diam}(y)$ centered on the prediction of the same joint obtained in the previous stage and normalized so as to adjust the training in the current stage according to the model of the previous stage.

Next, the training data is then augmented by using multiple normalizations for each image and joint. Instead of using only the predictions from the previous stage, we generate simulated predictions. This can be achieved by randomly replacing the true positions of the joint $i$ with vectors sampled randomly from a two-dimensional normal distribution $N_i^{(s-1)}$, with means and variances equal to those of the displacements $\left(y_i^{(s-1)}-y_i\right)$ observed in all examples of the training data. The complete augmented training data can be defined by: firstly, sample an example and a joint uniformly from the original data, and then generate simulated predictions based on the displacement $\theta$ sampled from $N_i^{(s-1)}$.

$\begin{aligned} D_A^s= & \left\{\left(N(x ; b), N\left(y_i ; b\right)\right) \mid\left(x, y_i\right) \sim D, \delta \sim \mathrm{N}_i^{(s-1)},\right. \\ & \left.b=\left(y_i+\delta, \sigma \operatorname{diam}(y)\right)\right\}\end{aligned}$                          (6)

The training objective of the cascade stage S is shown in Eq. (4), with special attention to using the correct normalization for each joint.

$\theta_{\mathrm{s}}=\arg \min _\theta \sum_{(x, y) \in D_A^s}\left\|y_i-\psi_i(x ; \theta)\right\|_2^2$                  (7)

3.5 Feature description and determination of dance movement

Three elements are usually considered when evaluating the dance movements [18]: timing, amplitude, and direction.

Timing: any movement, can be broken down into three parts: static, dynamic, and static. The dance movement of the “start and finish” must be at the right timing for a higher evaluation;

Amplitude: the amplitude of the movements should be basically consistent with the standard (here refers to the standard library), since persons vary in height.

Direction: it’s not required strictly. The right direction can make the front and back of the action linked more smoothly, thus resulting in higher evaluations.

Since errors are inevitable, we need to consider how to define the criterion of "tolerable error". This relies on a large sample of dancers' movements and supervised training by experts. In local practice, 5% variation is acceptable, or it’s difficult to distinguish, e.g., if jumping jack requires a knee angle of no less than 90 degrees, then a range of 85° or more is considered acceptable (in fact, larger deviations may be acceptable, depending on the overall postural finish at the time - potentially masking minor details).

The above evaluations are highly dependent on expert experience. Combining the human movement as well as the structural characteristics of the human body, this paper chooses the 20 nodes defined by Deng et al. [22] to represent the dance movements, as shown in Figure 3.

3.png

Figure 3. The 20 nodes defined and numbered by Kinect

Twenty skeletal joint points are extracted, to constitute the first 20 feature values of the feature vector T for the joint coordinate point, and then a (quantitative) standardized description of a specific action is performed for these 20 feature values. These features can be expressed as:

$T_i=P_i\left(x_i, y_i, z_i\right), i=1,2 \ldots, 20$                   (8)

where, $x_i$ represents the projection of the point $P_i$ on the $x$ coordinate axis; $y_i$ is the projection of $P_i$ on the $y$ coordinate axis; $z_i$ is the one on the $z$ coordinate axis. The lens location is the origin of the coordinates. Assuming that the adjacent standard pose feature point is set as $\left(x_n, y_n, z_n\right)$, and the imitation pose feature point is $\left(x_m, y_m, z_m\right)$, then the feature vector values corresponding to the two pose feature points are

$n=\left(x_0, y_0, z_0\right)-\left(x_1, y_1, z_1\right)$            (9)

From Eq. (6), the set of feature vectors can be derived as $P=\left\{n_0, n_1, n_2, \ldots, n_8\right\}$, which is called the pose feature descriptor to describe the human action pose, as shown in in subgraph (a) of Figure 4.

4.0.png

Figure 4. Pose calibration and determination

Once a standard pose model is defined, the actual pose of the current test subject can be compared with the standard. As shown in subgraph (b) of Figure 4, if exceeding the tolerance error, the test pose can be considered to be non-standard, i.e., there are two problems with the movement: first, the angle between the thigh and the calf is not 120°; then, the calf is not perpendicular to the ground, but has an inward tilt. Therefore, it’s determined that such action needs to be corrected.

3.6 Pose matching of human movement

After acquiring the feature descriptors of the human movements, the matching and recognition problem between two poses is transformed into the discrepancy of feature descriptor. Regarding this problem, it can be judged by comparing the angle values between the feature vectors corresponding to the standard pose and the mimic pose descriptors. Suppose the feature vectors of the standard pose and the imitation pose feature descriptor are set as $a\left(x_{s t}, y_{s t}, z_{s t}\right)$ and $b\left(x_{r e}, y_{r e}, z_{r e}\right)$, respectively, then the cosine value of the angle value between the two poses is:

$\cos _{a i}=\frac{x_{s t} \cdot x_{r e}+y_{s t} \cdot y_{r e}+z_{s t} \cdot z_{r e}}{\sqrt{x_{s t}{ }^2+y_{s t}{ }^2+z_{s t}^2} \cdot \sqrt{x_{r e}{ }^2+y_{r e}{ }^2+z_{r e}^2}}$              (10)

In Eq. (6), $a_i$ is the angle values in the range of $\left[0,180^{\circ}\right]$. From this, the set of angle values can be derived as $\theta=$ $\left\{a_0, a_1, a_2, \ldots, a_8\right\}$, and the difference between the two poses can be expressed in this way. As a result, the sum of the angle values is given as:

$a_{s u}=\sum_{i=0}^8 a_i$             (11)

The weight of each angle value is calculated as:

$\omega_i=\left(1-e-\frac{a_i}{a_{s u}}\right) /\left(\sum_{i=0}^8 1-e-\frac{a_i}{a_{s u}}\right)$              (12)

Based on the Eq. (12), a set of weight values can be derived as $W=\left\{\omega_0, \omega_1, \cdots, \omega_8\right\}$, so as to obtain the normalized angle values:

$D=\sum_{i=0}^8 a_i, \omega_i$             (13)

The normalized angle value parameter D derived from Eq. (13) can represent the degree of difference between the standard pose and the imitation pose. On this basis, the accuracy of matching between the two poses can be identified by introducing the D value into the formula of pose matching, which is given by:

$S= \begin{cases}f\left(a_{\text {max }}\right) \cdot\left[\left(D_{s t}-D\right) \cdot \frac{100-S_{s t}}{D_{s t}}+S_{s t}\right] & , 0 \leq D \leq D_{s t} \\ 0 & , D>D_{s t}\end{cases}$                  (14)

where, S represents the recognition accuracy of pose matching, in the value range of $[0,100] ; D_{s t}$ is the threshold of the preset standard angle difference; $S_{s t}$ is the preset baseline matching parameter; $f\left(a_{\max }\right)$ is the limb offset limit function, which mainly benefits from the penalty factor of pose matching accuracy. The larger the value of $S$, the better the match between the two poses; the smaller the value $D_{s t}$, the more severe the match accuracy between the two poses. The specific calculation formula is given as:

$f\left(a_{\text {max }}\right)= \begin{cases}1-\frac{0.3}{M^2} \cdot a_{\max }^2 & , 0 \leq a_{\text {max }} \leq \sqrt{10 / 3 M} \\ -\frac{0.3}{M^2} \cdot a_{\max }^2 & , a_{\max }>\sqrt{10 / 3 M}\end{cases}$                 (15)