An Intelligent Safety Monitoring Approach for Multi-Person Behavior Recognition in Elderly-Child Shared Spaces Based on Video Imagery

In elderly-child shared spaces, elderly people move slowly and children behave actively. The behavioral features of the elderly and children are significantly different, and complex situations such as limb overlapping and scene occlusion are likely to occur when multiple targets are active simultaneously. These lead to difficulties for traditional single-target recognition algorithms to accurately distinguish the behavioral features of different objects, and multi-target tracking is also prone to feature confusion due to occlusion. At the same time, the annotated data of such scenes is often limited, especially lacking behavioral samples of elderly and children at different scales. Existing models tend to overfit on limited data and are difficult to learn robust behavioral patterns. In addition, the specific behaviors of elderly and children require more delicate semantic features to support recognition, while the feature richness extracted by conventional backbone networks is insufficient to meet the demand for precise recognition.

To address this, this paper proposes a multi-scale contrastive fine-tuning network for multi-target behavior recognition in elderly-child shared spaces. The model network structure is shown in Figure 1. The specific implementation approach is closely centered on the scene characteristics of elderly-child shared spaces. First, deep deformable convolution is used to optimize the backbone network, enhancing the network’s ability to capture behavioral features of different scales and types, and extracting richer semantic features to adapt to the significant differences in movement amplitude and behavior patterns between the elderly and children. Second, object-level features are incorporated during the training stage to supplement the number of positive samples of elderly and child targets at different scales, solving the problem of scarce specific behavior samples in such spaces, enabling the model to learn more robust prior knowledge from the enhanced original features, and improving the stability of identifying specific behaviors of the elderly and children. Third, an improved supervised contrastive branch is introduced to enhance the feature discriminability among different behavior categories, combined with a balanced fine-tuning strategy to balance the training weights of elderly and child behavior samples, ultimately improving classification accuracy on a limited image set, while enhancing the model’s anti-overfitting ability and ensuring stable and precise multi-target behavior recognition in real elderly-child shared spaces.

1.png

Figure 1. Network structure of a multi-target behavior recognition model for elderly-child shared spaces

The proposed model is based on the FasterRCNN framework, and layered improvements are made according to the characteristics of elderly-child shared spaces, forming a complete architecture of “feature extraction — region proposal — feature optimization — loss calculation.” In the feature extraction layer, the convolution operators of the traditional backbone network are replaced with deformable convolution operators to build an optimized ResNet network. This improvement enhances the network’s ability to resist complex interferences in elderly-child shared spaces, such as furniture occlusion and lighting changes, while improving few-shot classification performance, adapting to the characteristics of scarce and significantly different behavior samples of elderly and children in this scene, and providing more robust base features for subsequent recognition. In the region proposal layer, an auxiliary positive sample enhancement branch module is integrated into the region proposal network structure to adaptively capture the multi-scale object-level features of elderly and children, supplementing the number of positive samples at different scales, enabling the model to learn multidimensional prior knowledge of elderly and child behaviors from the base class during the basic training stage and improving the accuracy of candidate region generation. In the feature optimization and loss calculation layer, the model encodes candidate features through an improved spatial pyramid pooling network and incorporates supervised contrastive learning ideas to improve the loss function. Aiming at the characteristics of small intra-class variation and significant inter-class differences in behaviors in elderly-child shared spaces, the loss function strengthens instance-level intra-class compactness and inter-class separability by measuring the similarity between target encodings, allowing the model to more accurately distinguish various behaviors of elderly and children. In the training phase, an improved balanced fine-tuning method is introduced, which dynamically adjusts the training weights of elderly and child behavior samples to alleviate the problem of sample imbalance that may occur in few-shot scenarios, ensuring that the model can stably learn the behavioral features of the two groups in a limited dataset.

2.1 Optimized ResNet network

The core motivation for constructing the optimized ResNet network is to address the adaptation deficiencies of traditional ResNet in multi-target behavior recognition within elderly-child shared spaces. In such spaces, behavior samples are scarce and scenes are complex. The 3×3 traditional convolution used in ResNet has inherent limitations: independent convolution kernels per channel lead to weak feature correlation; translation invariance simplifies computation but limits the receptive field; extracted features are single and fail to capture the differentiated behavior details of elderly and children, seriously affecting detection accuracy in few-sample scenarios. Therefore, for the multi-target recognition needs of such spaces, the feature extraction mechanism of the backbone network must be reconstructed.

The optimized ResNet network is constructed based on the original ResNet structure with targeted modifications: all 3×3 traditional convolution modules are removed and uniformly replaced with deformable convolution, while the 1×1 convolution blocks at the head and tail of the main structure are retained. The retention of the 1×1 convolution blocks is to maintain channel mapping and feature fusion functions, ensuring dimensional matching of input and output features; replacing 3×3 convolution aims to break the limitations of traditional convolution and enhance feature extraction capabilities through the properties of deformable convolution, making it more adaptable to the recognition needs of multi-scale targets and complex behaviors in elderly-child shared spaces.

Specifically, the operational mechanism of deformable convolution is closely designed around the feature capture requirements of elderly-child shared spaces. For the input feature map A∈R^G×Q×Z, the feature vector A_uk∈R^Z at a pixel (u,k) is first extracted, and then linearly transformed by the Ω function into a J²-dimensional feature vector and unfolded into a J-dimensional kernel matrix G_uk. Assuming that the fully connected operation is denoted by FC(·), the ReLU activation function by RELU(·), and batch normalization by BN(·), then:

$G_{u k}=\Omega(A)=F C(B N(R E L U(F C(A))))$                    (1)

This process transforms the channel dimension information into spatial kernel information, allowing each pixel’s convolution kernel to be associated with pixel features within the surrounding [J/2] range. By performing point-wise multiplication, the channel information of the initial pixel is spread to the nearby spatial region, thereby collecting contextual information from a broader receptive field. This mechanism can effectively capture the spatial correlation of targets in elderly-child shared spaces, such as limb interaction features between elderly and children, compensating for the insufficient receptive field of traditional convolution. Let the output feature after deformation be B, then:

$\left[a_1, a_2, \ldots, a j^2\right] \in \Re^{Z \times J^2} \otimes G_{u k}=\left[b_1, b_2, \ldots, b j^2\right]=S(a)$                          (2)

The final output is:

$B=\sum_{u=1}^{j^2} b_u$                  (3)

The construction of the optimized ResNet network provides key support for multi-target behavior recognition in elderly-child shared spaces. Through channel dimension shared kernels, the deformable convolution performs multiplication and addition operations in a sliding window manner, summarizing context in a broad spatial structure, significantly expanding the receptive field and enhancing the feature capturing ability for elderly and child targets in complex scenes. At the same time, this design reduces parameter redundancy, improves feature correlation and richness, enables the learning of more robust behavior patterns from limited samples, effectively alleviates the overfitting problem in few-sample scenarios, and provides more precise high-level semantic features for subsequent region proposal and feature optimization, aiding efficient recognition of multi-target behaviors in elderly-child shared spaces. Figure 2 shows the schematic diagram of the deformable convolution in the optimized ResNet network.

2.png

Figure 2. Schematic diagram of deformable convolution in the optimized ResNet network

2.2 Positive sample enhancement

The design of the positive sample enhancement branch module originates from the need to fully utilize object-level features in multi-target behavior recognition within elderly-child shared spaces. The network architecture is shown in Figure 3. In such spaces, the body size difference between elderly and children is significant, and frequent multi-target interactions easily lead to feature confusion. Meanwhile, the limited sample data contains insufficient positive samples at each scale, making it difficult for the region proposal network to accurately locate target regions. To solve this problem, the module enhances the positive sample features at each scale, strengthens the model’s attention to elderly and child target regions, and compensates for the deficiencies of traditional region proposal networks in multi-scale target localization. This provides a more precise regional basis for subsequent behavior recognition and aligns with the research objective of improving feature enhancement effect and recognition accuracy.

3.png

Figure 3. Network structure of the positive sample enhancement branch module

The structural design of this module closely fits the multi-scale target characteristics of elderly-child shared spaces. First, the ground truth objects in the input image are cropped into multiple sizes from 32×32 to 800×800 to match the anchor sizes of each feature layer in the FPN, ensuring coverage of different body size scales of elderly and children in space and providing suitable inputs for positive sample feature enhancement at each scale. Second, after feature extraction via ResNet to different feature stages p2~p6, the corresponding scale feature maps are processed in two paths: one path is sent into the region proposal network, after 3×3 convolution and 1×1 convolution, used to calculate the foreground feature matrix of positive samples, which is aggregated and superimposed with the candidate feature matrix extracted by the optimized ResNet network to improve the object score of elderly and child targets; the other path enters the spatial pyramid pooling network, is downsampled to 14×14, and fused with the filtered proposal regions, then input into the detection head for decoding, enhancing the feature information of positive samples at each scale. This dual-path design not only optimizes the quality of proposal boxes but also strengthens feature correlation, adapting to the complex distribution of multi-scale targets in elderly-child shared spaces.

The positive sample enhancement branch module provides strong support for the model’s multi-target behavior recognition through a targeted feature enhancement mechanism. By matching the cropped positive samples at different scales with anchors and integrating features, the module effectively supplements the scarce multi-scale positive sample features in elderly-child shared spaces, enabling the network to more accurately locate behavior regions of different scales such as child climbing or elderly standing up, and reducing localization deviation caused by target scale differences. At the same time, the aggregation and superposition of the foreground feature matrix and candidate feature matrix strengthen the feature saliency of elderly and child target regions, reduce the impact of background interference and target occlusion, provide more reliable feature input for subsequent behavior recognition modules, support the model to achieve more efficient multi-target behavior recognition in complex scenarios, and further promote the research objectives of feature enhancement and accurate recognition.

2.3 Loss function design

The construction of the supervised contrastive loss originates from addressing the problem of classification confusion of multi-target behaviors during few-shot fine-tuning in elderly-child shared spaces. In such spaces, some behavioral features of elderly and children targets exhibit similarity, and the sample data is limited, causing the model to easily confuse different categories of behaviors during fine-tuning. Traditional loss functions are difficult to accurately characterize instance-level differences in elderly and child behaviors, while supervised contrastive loss, inherited from contrastive learning ideas, can help the model learn more distinguishable high-level semantic features by strengthening intra-class feature aggregation and inter-class feature separation. This aligns with the research goal of improving multi-target behavior recognition accuracy under few-shot conditions. Its specific construction method is closely centered around behavioral features in elderly-child shared spaces. In feature processing, the proposed box features extracted by the spatial pyramid pooling network are flattened and mapped through a fully connected layer to a 1024-dimensional feature vector p, and then L2-normalized and encoded to reduce the dimension to a 128-dimensional embedding feature p^~, which simplifies the comparison complexity of elderly and child behavior features and adapts to the feature dimension requirements of multi-target behavior in this space. In similarity measurement, the cosine projection space is used to compute the similarity between embedding features to characterize the class belonging probability of elderly and child behaviors. That is, for significantly different behaviors such as child climbing and elderly falling, the similarity of different-class features is reduced; for same-class behaviors, the similarity is increased, clarifying intra-class compactness and inter-class separability. In loss function design, when the total similarity of same-class features is larger, the loss becomes smaller; when the total similarity of different-class features is smaller, the loss also becomes smaller, guiding the model to focus on behavioral differences of multi-targets in elderly-child shared spaces. Figure 4 shows a schematic diagram of the loss function design principle. Assume that any o^~_u and its same-class features are represented by o^~_k, and different-class features are represented by o^~_j; the total number of embedding features o^~ is denoted as V, the ground truth corresponding to feature o^~_u is denoted as b_u, the number of features of b_u is V_bu, and δ is a hyperparameter greater than P. Based on the above, the loss functions are designed as:

$\operatorname{LOSS}_{E X}=\frac{1}{V} \sum_{u=1}^V \operatorname{LOSS}_{o u}$                          (4)

$\begin{aligned} & \operatorname{LOSS}_{\text {ou }} =\frac{-1}{V_{b u}-1} \sum_{k=1, k \neq u}^V I I\left\{b_u=b_k\right\} \cdot \log \frac{\exp \left(\tilde{O}_u \cdot \frac{\tilde{o}_k}{\delta}\right)}{\sum_{j=1}^V I I_{j \neq u}}  \cdot \exp \left(\tilde{O}_u \cdot \frac{\tilde{O}_j}{\delta}\right)\end{aligned}$                    (5)

4.png

Figure 4. Schematic diagram of the loss function design principle

In summary, the total loss function of the model is defined as consisting of the binary cross-entropy loss LOSS_EOV for optimizing foreground objects, the binary cross-entropy loss LOSS_zmt for foreground object classification, the smooth L1 loss LOSS_REG, and LOSS_EX:

$\begin{aligned} L O S S_{\text {total }} & =L O S S_{E O V}+L O S S_{z m t}  +L O S S_{R E G}+\frac{1}{2} L O S S_{E X}\end{aligned}$                (6)

The supervised contrastive loss plays a key role in the model. It, together with the binary cross-entropy loss and smooth L1 loss, constitutes the total loss function to jointly optimize the model performance. In few-shot scenarios, it can effectively reduce the classification confusion of elderly and child behaviors, such as avoiding misjudging elderly people sitting down slowly as children squatting, and enhance the model’s ability to distinguish similar behaviors. Meanwhile, by prompting the model to focus on features of the same-class behavior, it strengthens the semantic discriminability of multi-target behavior features in elderly-child shared spaces, provides loss constraints for the model to learn robust behavior patterns from limited samples, and helps to achieve the research goals of feature enhancement and precise recognition, thereby improving the overall reliability of multi-target behavior recognition.

2.4 Model training

This paper introduces a balanced fine-tuning method to solve the problem of sample imbalance in deep network training under few-shot conditions in elderly-child shared spaces, aiming to improve the accuracy of multi-target behavior recognition under few-shot scenarios. In such spaces, behavior samples specific to elderly and children are often scarce, while common behavior samples are relatively sufficient. During fine-tuning, new-class positive samples are likely to be erroneously suppressed by non-maximum suppression due to low foreground confidence in the region proposal network. At the same time, excessive background negative samples interfere with the model’s learning of effective features. Traditional fine-tuning strategies cannot balance the training weights of base classes and new classes, making it difficult for the network to accurately capture specific behavioral features of the elderly and children. Therefore, a targeted balanced mechanism is needed to optimize the training process.

To address the problems of positive sample filtering and negative sample interference, the balanced fine-tuning method achieves optimization through joint training and sample selection strategies. The region proposal network and the spatial pyramid pooling network feature extractor are jointly trained under elderly-child shared space object supervision, which can double the number of candidate boxes that pass non-maximum suppression, ensuring that rare new-class positive samples such as elderly falling and child climbing are retained. Meanwhile, the number of candidate boxes used for loss calculation in the spatial pyramid pooling network is halved to reduce interference from background-only negative samples, enabling the model to focus more on effective behavior features of elderly and child targets. Assume that the original gradients obtained from training the new class and base class are h_v and h_y, and the updated gradient values are h^~, h^~_y. The following formulas give the optimization objective function of fine-tuning training:

$\operatorname{MIN} S\left(\tilde{h}_y, \tilde{h}_v\right)=\frac{1}{2}\left\|h_v-\tilde{h}_v\right\|_2^2+\frac{1}{2}\left\|h_y-\tilde{h}_y\right\|_2^2$                     (7)

$\tilde{h}_v=h_v-\left(\frac{h_v^S \cdot h_y}{h_y^S \cdot h_y}\right) \cdot h_y, \tilde{h}_y=h_y-\left(\frac{h_y^S \cdot h_v}{h_v^S \cdot h_v}\right) \cdot h_v$                  (8)

In elderly-child shared spaces, the base class data is large while the new-class samples are scarce. Traditional gradient updates tend to favor base classes, making it difficult for the network to remember new-class features of specific elderly and child behaviors. Balanced gradients can adaptively reweight the gradients of the two types of data, allowing the network to quickly acquire new-class information from limited samples and efficiently generalize to various behaviors of elderly and child groups. Ultimately, this improves the stability and accuracy of the model's multi-target behavior recognition under few-shot conditions. Specifically, this paper adopts balanced gradient descent instead of traditional stochastic gradient descent, computing the weighted average of base class gradient h_y and new class gradient h_v as the balanced gradient h^~, with the calculation formula as follows:

$\tilde{h}=\frac{1}{2}\left(1-\frac{h_v^S \cdot h_y}{h_y^S \cdot h_y}\right) \cdot h_y+\frac{1}{2}\left(1-\frac{h_y^S \cdot h_v}{h_v^S \cdot h_v}\right) \cdot h_v$                 (9)

From the experimental data comparison in Table 1 and Table 2, it can be seen that on both the training set and the test set, the method proposed in this paper achieves significantly better average detection accuracy for new classes under single-target and multi-target scenarios compared with the baseline models including Mask R-CNN, Model-Agnostic Meta-Learning, SENet, NAS-FPN, and Feature Reconstruction Detector. Specifically, in the training set, the accuracy of Our Method reaches 62.8% in the 4-target scenario, far exceeding the 41.5% of Mask R-CNN. In the test set, ours achieves an accuracy of 52.3% in the 4-target scenario, also outperforming other models. This indicates that by optimizing the feature extraction network, the proposed method enhances the discriminability of elderly and child behavior features, making feature extraction and recognition more robust in complex multi-target scenarios, effectively improving the accuracy of multi-target behavior detection. The multi-target accuracy in both the training and test sets steadily increases with the number of targets and always remains higher than that of the comparison models. In summary, the experimental data fully verify the effectiveness of the proposed method in multi-target behavior recognition: it performs excellently not only in single-target scenarios but also shows stronger detection capability in complex multi-target elderly-child shared scenes, significantly surpassing existing comparison methods and providing high-precision and robust technical support for safety monitoring in elderly-child shared spaces.

Table 1. Experimental data comparison on training set / %

Table 2. Experimental data comparison on test set / %

Through the ablation experiment analysis in Table 3, the proposed method demonstrates significant effectiveness under the collaborative optimization of multiple modules. In the experiment, No. 1 represents the baseline without any optimization modules, and its average detection accuracy for new classes is low under both single and multi-target scenarios, reflecting the insufficient capability of the baseline model to extract and distinguish behavior features of elderly and children. No. 2 only applies the optimized ResNet network, with a slight improvement in accuracy, verifying the fundamental role of the optimized feature extraction network in behavior recognition. After adding the positive sample enhancement branch in No. 3, the accuracy increases significantly, indicating that the enhancement of positive samples effectively strengthens the discriminability of elderly and child behavior features and improves the robustness of recognition in complex multi-target scenarios. No. 4 further introduces supervised contrastive loss, with accuracy continuing to improve, reflecting the continuous optimization of model performance through module collaboration. The experimental results show that the feature-enhanced multi-target behavior recognition framework constructed through the optimized ResNet network, positive sample enhancement branch, and supervised contrastive loss, demonstrates layered progression and synergistic improvement across modules, significantly enhancing detection accuracy in both single and multi-target scenarios. The ablation experiments fully validate the effectiveness of the proposed method: the multi-module optimization based on feature enhancement greatly improves the accuracy and robustness of multi-target behavior recognition.

Table 3. Ablation experiments of each module

Through the trend analysis of Loss and mAP in Figure 5, the proposed method shows significant performance optimization and convergence characteristics during training, fully validating its effectiveness. In the left Box subplot, the bounding box regression loss rapidly decreases with iterations and stabilizes, indicating that the model's ability to predict the positions of elderly and child targets is continuously improved under the effect of feature enhancement technology, accurately locating multi-target behavior subjects in complex scenes, and providing reliable spatial location information for subsequent safety monitoring. In the middle Classification subplot, the rapid decline of classification loss reflects the model's significantly enhanced ability to identify elderly and child behavior categories, thanks to the strengthening of feature discriminability, enabling the model to accurately classify behavior types in elderly-child scenarios with coexisting multiple targets, providing key semantic information for safety risk assessment. In the right mAP@0.5 subplot, the mean average precision quickly rises and reaches a high value close to 0.9, reflecting the effect of collaborative optimization between Box and Classification loss, proving that the overall performance of the proposed method in multi-target behavior recognition far exceeds that of the baseline model.

5.png

Figure 5. Trends of loss and mAP @ 0.5 of the proposed method

Through the analysis of the confusion matrix in Figure 6, the proposed method shows excellent performance in multi-target behavior recognition in elderly-child shared spaces. The diagonal elements of the matrix are all at high levels, such as "supporting" 0.96, "assisting" 0.96, "playing" 0.91, "falling" 0.88, indicating significant correct recognition rates for core behaviors including safe interaction, playing, and high-risk abnormal behaviors, and reflecting the effectiveness of feature enhancement technology in distinguishing elderly and child behavior features. For example, the high recognition rate of positive safety behaviors such as “supporting” and “assisting” validates the model’s accurate capture of elderly-child mutual aid scenarios; the accurate classification of high-risk behaviors such as “falling” and “pushing” ensures timely warning of abnormal behaviors, providing a reliable behavior recognition basis for safety monitoring strategies. Among the off-diagonal elements, although there are some misclassifications, the overall error is controllable, and the recognition accuracy of key safety-related behaviors still meets practical application requirements. For example, “falling” as an emergency risk behavior, with an accuracy rate of 0.88, can effectively trigger emergency responses. Combined with the research content, the feature enhancement module improves the robustness of the model in complex scenarios by strengthening the discriminability of behavior features, making the confusion matrix show a desirable distribution of "high diagonal, low off-diagonal errors", providing high-precision behavior classification results for the safety risk assessment model. In summary, the confusion matrix visually reflects the high recognition accuracy of the proposed method for multi-target behaviors in elderly-child shared scenarios, especially for precise classification of safety-related behaviors, fully proving the collaborative effectiveness of the technical approach and effectively improving the safety monitoring capability of elderly-child shared spaces.

6.png

Figure 6. Confusion matrix

Through the analysis of the PR curve in Figure 7, the proposed method shows outstanding performance in multi-target behavior recognition in elderly-child shared spaces. The PR curves of various behavior categories are close to the upper left corner, indicating that precision remains high under different recall rates. The average precision (AP) of safe interaction behaviors such as "playing", "assisting", "organizing", as well as high-risk behaviors such as "pushing" and "falling", all exceed 0.9, reflecting the precise extraction and discrimination capability of the feature enhancement technology for elderly and child behavior features. The overall mAP@0.5 of all categories reaches 0.911, further verifying the model’s comprehensive recognition performance under multi-target scenarios. By optimizing the feature extraction network, the model is able to classify safe interaction and risky behaviors with high accuracy in complex elderly-child coexisting scenarios, providing a reliable behavior recognition basis for safety monitoring strategies. Based on the high-accuracy behavior recognition results, the safety risk assessment model can capture abnormal behaviors in real time and trigger emergency responses, achieving proactive safety prevention and control. The excellent performance of the PR curve fully proves the effectiveness of the proposed method: from feature enhancement to behavior recognition, the technical optimization enables the model to achieve both accuracy and robustness in multi-target behavior detection in elderly-child shared spaces, strongly supporting the implementation of safety monitoring strategies and providing a practical solution for the safety protection of elderly and child groups.

7.png

Figure 7. PR curve