Image-Based Intelligent Classroom Monitoring and Learning Behavior Analysis via Joint Object Detection and Semantic Segmentation

2.1 Model design

According to the needs in the intelligent classroom environment, this paper needs to detect whether students are studying attentively in their seats, and at the same time determine whether workers have left the learning area, which requires the detection of the boundary lines of the intelligent classroom environment. The former belongs to the object detection task, while the latter can be regarded as a semantic segmentation task. In order to meet real-time requirements, this paper designs an intelligent classroom environmental monitoring and learning behavior detection model. The model adopts a structure with shared encoder and dual decoder branches, mainly based on the dual needs of "collaboration between object detection and semantic segmentation" and "real-time performance assurance" under the intelligent classroom scenario. On the one hand, learning behavior detection belongs to object detection, which requires locating specific objects and judging their states; intelligent classroom environmental monitoring belongs to semantic segmentation, which requires pixel-level boundary division. If two independent models are used for processing respectively, it will not only cause repeated computation in the feature extraction process and increase resource consumption, but also reduce the consistency of the results due to the disconnection of the feature correlation between the two tasks. For example, spatial matching deviations may occur between student position judgment and boundary line segmentation. The shared encoder can generate shared low-level features for both tasks through one-time feature extraction, while the dual decoder branches respectively perform feature refinement according to the task characteristics of object detection and semantic segmentation. This can reduce redundant computation to meet the response speed requirements of real-time monitoring, and also enhance the spatial correlation between “student status” and “region boundary” through feature sharing, thus improving the coordination of overall detection.

The BiFPN used in the model introduces a weight allocation mechanism similar to the attention mechanism to solve the problem of accuracy in detecting dim boundary lines in intelligent classrooms. In actual classroom scenarios, boundary lines such as the junction between the wall and the floor, and the separation line between the learning area and the non-learning area often appear dim due to uneven lighting, equipment occlusion and other factors. These boundary lines are the core basis for judging whether workers have left the learning area. If boundary line detection is blurry or broken, it may lead to incorrect region division and thus misjudgment. Traditional attention mechanisms often assign weights based on global feature distribution, which tends to suppress features in low-contrast regions such as dim boundary lines. The improved attention mechanism dynamically adjusts the weight allocation of feature channels and spatial positions to enhance feature response to dim regions: strengthening the feature channels related to edge detection in the channel dimension, and focusing on low-brightness but continuous edge regions in the spatial dimension, thereby improving the completeness and clarity of boundary lines and providing reliable spatial basis for region judgment.

The model also innovatively adopts an improved version of EFL focal loss, which directly addresses the core contradiction of “pixel imbalance” in intelligent classroom boundary line detection. As slender structures, boundary lines usually account for less than 5% of pixels in images, while background pixels account for an extremely high proportion, forming a typical long-tail distribution. If traditional loss functions are used, the model will overfit background features due to the numerical advantage of background pixels, causing the training weight of boundary line pixels to be diluted, and resulting in broken or missing boundary lines in the final segmentation result. This will directly affect the accurate definition of the learning area, thereby interfering with the judgment of "whether workers have left the learning area." The improved EFL focal loss solves this problem through two key optimizations: first, an adaptive hard sample mining mechanism is introduced to assign higher loss weights to “hard-to-detect pixels” such as boundary lines, ensuring that the model focuses on low-proportion but key regions during training; second, the loss ratio of positive and negative samples is dynamically adjusted to avoid excessive suppression of boundary line loss signals by background pixel loss values. This design enables the model to balance the learning priority of different pixel categories during training, ultimately improving the accuracy and stability of boundary line segmentation. Figure 1 shows the structure diagram of the intelligent classroom environmental monitoring and learning behavior detection model.

93d36ef0-f7e3-4a4c-a284-f44274426e6f.png

Figure 1. Structure diagram of intelligent classroom environmental monitoring and learning behavior detection model

2.2 Model structure

The algorithm used in this paper is an improved version based on YOLOP. In order to make the network characteristics highly compatible with the requirements of object detection and semantic segmentation in intelligent classrooms, this paper chooses CSPNet as the backbone network. To ensure that the model can simultaneously perform object detection and semantic segmentation, the backbone network is required to extract sufficiently rich features while meeting the efficiency requirements of real-time monitoring. CSPNet happens to meet both needs: on the one hand, as an efficient structure verified by mainstream detection models such as YOLOv4 and YOLOv5, it inherits CSPNet's advantage of "enhancing feature representation capability", and can extract multi-level features from classroom images, including texture, shape, and edge. These features can support both object detection in judging student positions and states, and semantic segmentation in providing edge features for boundary line recognition; on the other hand, CSPNet achieves "lightweight and efficiency" through gradient flow optimization, with significantly lower computation and parameter count than traditional networks, enabling improved inference speed while ensuring feature richness, which aligns with the scenario requirements of real-time monitoring in intelligent classrooms. Figure 2 shows the structure diagram of the CSPNet used.

8baa4581-6e7f-4cce-9490-4c1705805d9c.png

Figure 2. Structure diagram of the CSPNet used

The core principle of CSPNet in supporting object detection and semantic segmentation comes from its "gradient flow optimization" and "feature fusion design". At the gradient flow level, CSPNet divides the basic layer feature map into two parts, and realizes separated fusion through cross-stage layers, which avoids the optimization bottleneck caused by repeated gradient information in traditional networks, allowing gradients to propagate along different paths. This not only retains the richness of features but also reduces redundant computation. At the feature extraction level, CSPNet inherits the advantage of “feature reuse” from DenseNet, while avoiding feature redundancy through truncated gradient flow. The generated feature maps possess both “detail preservation” and “global consistency”, which precisely match the feature requirements of object detection and semantic segmentation in intelligent classrooms: object detection requires accurate local features to locate students, while semantic segmentation requires continuous global features to recognize boundary lines. In addition, CSPDarknet, as a fusion of Darknet53 and CSPNet, further enhances the lightweight property and robustness. Its multiple CSP modules can progressively extract features from low-level to high-level, providing basic features adapted to different tasks for the subsequent dual decoder branches. Figure 3 shows the backbone network structure diagram of the constructed model.

ac5537d1-355a-47dc-bc5c-fd4fd85f8540.png

Figure 3. Backbone network structure diagram of the constructed model

From the task characteristics perspective, object detection for “whether students are studying attentively in their seats” needs to deal with targets of different scales, such as clear postures of front-row students and small-scale silhouettes of back-row students, while semantic segmentation for “classroom boundary line detection” needs to consider both fine-grained edges at the junction of desks and floors, and global region division of learning and non-learning areas. Both types of tasks rely on the effective fusion of multi-scale features. Therefore, this paper chooses BiFPN as the network Neck. BiFPN, as an improved version of PANet, happens to meet this requirement: its bidirectional cross-scale connection design can aggregate features of different levels output from the backbone network, including low-level texture features, mid-level shape features, and high-level semantic features, generating fusion features with both detail and global information. This provides multi-scale basis for judging student postures in object detection, and supplements continuous edge features of boundary lines for semantic segmentation. Meanwhile, BiFPN's weighted feature fusion mechanism and lightweight structure can control time cost while ensuring fusion effect, avoiding the decline of real-time performance due to complex feature processing, which fits the real-time monitoring scenario of intelligent classrooms, such as rapid response to dynamic changes in student behavior and environmental status during class. Figure 4 shows the structure diagram of the BiFPN used.

bce3a43f-bf4a-4d2b-a3d0-0f64f953f48e.png

Figure 4. Structure diagram of the BiFPN used

The basic principle of BiFPN in supporting object detection and semantic segmentation originates from the synergy of its “bidirectional feature flow” and “weighted fusion mechanism”. At the feature flow level, BiFPN realizes bidirectional complementarity of cross-scale features through repeated top-down connections from high-level semantic features to low-level detail features and bottom-up connections from low-level detail features to high-level semantic features: the top-down path can inject semantic information such as “learning area” and “student identity” into low-level features, improving the regional correlation of boundary line segmentation and better distinguishing between “student seat area boundary” and “non-learning area boundary”; the bottom-up path can integrate detail features such as “edge texture” and “local contour” into high-level features, enhancing the detection ability of object detection for small-scale students or ambiguous postures, including downward head movements of back-row students. At the fusion mechanism level, the weight allocation mechanism similar to attention mechanism introduced by BiFPN can dynamically adjust the contribution of features from different sources. For example, in boundary line detection, it enhances the weight of low-level edge features, and in student posture judgment, it increases the proportion of mid-level shape features, making the fused features more adapted to the requirements of specific tasks.

Specifically, from the scenario requirements of the intelligent classroom, the detection branch needs to perform object localization and classification of "whether students are attentively studying in their seats", which belongs to sparse object detection and requires attention to the position and category of discrete targets. The segmentation branch needs to realize pixel-level classification of "boundary line and background", which belongs to dense pixel classification and requires attention to the pixel attribution of continuous regions. If a single branch is used to handle both types of tasks, the conflict of optimization objectives will lead to a decline in accuracy for both tasks. Based on the essential differences in requirements between object detection and semantic segmentation tasks, this paper adopts a separated design with two independent decoder branches to ensure that each branch optimizes the task characteristics in a targeted manner and avoids "task interference" during the feature processing.

The design principle of the detection branch focuses on the demand of "sparse object multi-scale localization", realizing accurate detection through the PAN structure and anchor mechanism. In the intelligent classroom, students may appear in different positions such as the front row, back row, or corners, with significant differences in object scale. This requires the detection branch to have multi-scale adaptation capability. The bottom-up feature migration property of the PAN structure can transmit low-level precise positional features to the high level, compensating for the deficiency of traditional feature pyramids that have strong semantics but weak localization, and providing accurate spatial reference for locating targets at different scales. The multi-scale feature maps with 8×, 16×, and 32× downsampling can match small, medium, and large-scale objects respectively, and combined with the grid allocation strategy of three types of anchor boxes, can cover the common scale range of students in classroom scenarios. In addition, the design of prediction parameters such as position offset, scale transformation, and class probability directly corresponds to the core requirements of "locating student position" and "judging whether the student is studying attentively", making the detection results directly usable for the subsequent judgment of "whether the student is within the learning area". Specifically, assume that the predicted bounding box is represented by (ya, yb, yq, yg), the coordinates of the top-left corner of the grid are represented by (za, zb), and the width and height of the anchor box assigned to the grid are represented by (oq, og). The transformation formulas are as follows:

$y a=(2 * {SIGMOID}(s a)-0.5)+z_a$    (1)

$y b=(2 * {SIGMOID}(s b)-0.5)+z_b$     (2)

$y q=O_Q * e^{s q}$    (3)

$y g=O_g * e^{s g}$     (4)

${SIGMOID}(a)=\frac{1}{1+e^{-a}}$    (5)

The design principle of the segmentation branch is centered on "lightweight and efficient pixel-level classification", adapting to boundary line detection needs through a simplified process and targeted upsampling strategy. The core of boundary line detection in intelligent classrooms is to distinguish "boundary line pixels" from "background pixels", without requiring complex semantic category classification. Therefore, the segmentation branch adopts a lightweight design of "3 times upsampling + nearest neighbor interpolation": selecting the lowest 8× downsampled 32×32 feature map from the Neck as input, which retains basic edge features of boundary lines, avoids detail loss caused by excessively high downsampling rates, and controls the size of the initial feature map to reduce computation. Three times of upsampling restore the feature map to the size of the input image, ensuring that the output can correspond one-to-one with the pixels of the original image, meeting the accuracy requirements of "pixel-level segmentation". Nearest neighbor interpolation is chosen instead of deconvolution, significantly reducing computational complexity at the cost of a small amount of detail accuracy, avoiding delay in overall model real-time performance due to the segmentation process. The final output feature map of dimension (W, H, 2) directly provides the probability distribution of “boundary line/background”, which can provide clear pixel-level basis for “learning area division”, forming spatial correlation with the student position information from the detection branch.

2.3 Model loss function

The constructed model loss consists of the detection head loss and the segmentation head loss. The loss of the detection head mainly comes from classification loss (loss_CL), confidence loss (loss_OBJ), and bounding box regression loss (loss_BOX), while the boundary line segmentation head uses weighted cross-entropy loss. The expression of the loss function of the detection head is as follows:

$l o s s_{D E T}=\beta_1 l o s s_{C L}+\beta_2 l o s s_{O B J}+\beta_3 l o s s_{B O X}$    (6)

In intelligent classroom scenarios, samples of students "studying attentively" account for a very high proportion in the collected data, while samples of "not studying attentively" account for a very low proportion, forming a typical long-tail distribution. If traditional loss functions are used, the model will be dominated by the losses of majority class samples during training, leading to weak recognition ability for minority class samples. In order to solve the problem of “long-tail distribution of sample classes” in intelligent classroom learning behavior detection and achieve accurate balance of class loss, the classification and confidence loss adopt the design principle of EFL loss. EFL, as an improved version of Focal Loss, dynamically adjusts the loss weights of samples from different classes, which can suppress overfitting of majority classes and strengthen the loss signals of minority classes, making the model pay more attention to rare but important abnormal behavior samples during training. At the same time, using EFL for confidence loss can improve the reliability of judgment on “whether the student is in the seat”: for ambiguous samples such as students half-standing or occluded, EFL can force the model to focus on hard samples’ feature learning by reducing the weight of easy samples, so that classification and confidence results better match the core requirement of “recognizing learning behavior status”. Let x_s represent the balance between positive and negative samples, o_s represent the confidence score of the predicted target. The Focusing Factor in balanced data scenarios that controls the basic behavior of the classifier is represented by parameter ε_y. The cumulative gradient ratio of the positive and negative samples of class k is represented by parameter h^k, and the scaling factor is represented by hyperparameter t. The expression of the EFL loss function is as follows:

$E F L\left(o_s\right)=-\sum_{k=1}^z \beta_s\left(\frac{\varepsilon_y+\varepsilon_n^k}{\varepsilon_y}\right)\left(1-o_s\right)^{\varepsilon_y+\varepsilon_n^k} \log \left(o_s\right)$    (7)

$\varepsilon_n^k=t\left(1-h^k\right)$    (8)

The bounding box regression loss chooses Expected Intersection over Union (EIOU) and integrates the design principle of Focal Loss, aiming to improve the accuracy and convergence efficiency of student position localization, adapting to the detection needs of dynamic targets in intelligent classrooms. The key premise of learning behavior detection is accurate localization of students in their seats. The predicted bounding box must highly match the actual student position, which requires the regression loss to comprehensively measure the difference between the predicted box and the target box. EIOU is designed with three components: “overlap loss, center distance loss, width-height loss”. Compared with Complete Intersection over Union (CIOU), it more directly optimizes box shape and position. In intelligent classrooms, this can quickly reduce the gap between the predicted box and the actual position of the student and accelerate model convergence. After integrating Focal Loss, the loss function can reduce the optimization weight of low-overlap anchor boxes and focus training on high-overlap anchor boxes, further improving localization accuracy, which is crucial for “judging whether the student is in the seat”. Suppose the width and height of the minimum enclosing box covering both the predicted and ground truth boxes are denoted as z_q and z_g, the centers of the predicted and ground truth boxes are denoted by y and y^hs, ϑ calculates the Euclidean distance between the two centers, and the width and height of the predicted and ground truth boxes are represented by q, q^hs and g, g^hs respectively. The calculation formula for CIOU is as follows:

$\begin{aligned} & \operatorname{loss}_{R L P I}=\operatorname{loss}_{U P I}+\operatorname{loss}_{D I C}+\operatorname{loss}_{A S P} \\ & =1-I o U+\frac{\vartheta^2\left(y, y^h\right)}{\left(z_q\right)^2+\left(z_g\right)^2} \\ & +\frac{\vartheta^2\left(q, q^h\right)}{\left(z_q\right)^2}+\frac{\vartheta^2\left(g, g^{h s}\right)}{\left(z_g\right)^2}\end{aligned}$     (9)

By integrating EIOU Loss and Focal Loss, and letting λ be the hyperparameter that controls the curvature of the loss curve, the final EIOU Loss expression is:

$n l o s s_{F-E}=I o U^2 l o s s_{R U P I}$    (10)

The overall loss function design of the detection head serves the core goal of "accurate detection of learning behavior" through the synergy of "class balance" and "localization optimization". The EFL design of classification and confidence loss ensures that the model can effectively distinguish between "serious learning" and "non-serious learning" in a long-tailed sample distribution, avoiding missed detections of abnormal behaviors due to class imbalance. The EIOU+FocalLoss design for bounding box regression ensures the accuracy and stability of student position localization, providing reliable spatial reference for determining whether students are seated. The combination of both enables the detection head output to accurately reflect the core features of learning behavior and to link with the subsequent boundary line segmentation results, ultimately achieving the dual-precision monitoring of “learning behavior status + spatial position” in smart classrooms.

The segmentation head for boundary line detection adopts weighted cross-entropy loss, whose core principle is to specifically address the extreme imbalance between "boundary line pixels and background pixels" in the smart classroom scenario by forcing the model to focus on critical boundary line pixels through a weighting mechanism. In smart classroom images, boundary lines usually appear as thin elongated lines, accounting for less than 5% of the total pixels, while background pixels occupy an overwhelmingly large proportion. If the original cross-entropy loss is used, the model will overfit the background features due to the numerical advantage of background pixels, causing the classification errors of boundary line pixels to be diluted. This manifests as broken, blurred, or "swallowed" boundary lines in the segmentation results, directly affecting the accurate division of learning regions. The weighted cross-entropy loss assigns higher loss weights to boundary line pixels, significantly increasing their share in the total loss: when the model misclassifies boundary line pixels, it incurs higher loss values, forcing the model to enhance its learning of low-proportion boundary line features during training. This ensures the integrity and clarity of boundary line segmentation and provides reliable pixel-level support for distinguishing between “learning areas and non-learning areas”. Assuming the balance factor is represented by q_Z, the total number of pixels in the image is denoted as V, and the total number of pixels in the Z-th category of the foreground is denoted as V_Z, the loss function is expressed as:

$l o s s_{m m-S E G}=-\sum_{z=1}^L q_z b_z \log \left(o_z\right)$    (11)

$q_z=\frac{V-V_Z}{V}$    (12)

The design of this loss function further serves the overall research goal of "linking environmental monitoring with learning behavior analysis" by improving segmentation precision as a foundation for spatial relationship judgment. One of the core requirements in smart classrooms is to perform spatial inference by combining the "boundary line segmentation result" with the "object detection result", which demands high spatial accuracy in boundary line segmentation. If boundary line positions deviate or are missing, it directly leads to region segmentation errors, resulting in behavioral judgment deviations. Weighted cross-entropy loss, while balancing pixel loss, retains the natural suitability of cross-entropy loss for pixel-level classification: by minimizing the error between "network output probabilities of boundary line/background" and "ground truth pixel labels", it ensures each pixel's classification result closely reflects the actual scene.

Assuming the parameters used to balance the detection head and segmentation head losses are represented by δ₁ and δ₂ respectively, the overall loss function of the model is expressed as:

$l o s s_{A L L}=\delta_1 l o s s_{D E T}+\delta_2 l o s s_{m m-S E G}$    (13)

Figure 5 shows the flowchart for judging the smart classroom environment and student learning status.