Huali Chen*
| Minjing Peng | Xiaotian Du | Baisheng Lin | Mingjun Jiang | Jieyi Hu | Yuanxun Shao | Junyi Lin
© 2024 The authors. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
In the realm of public safety early warning monitoring, swiftly establishing an efficient abnormal behavior recognition model is of significant importance. We propose and implement a video-based abnormal behavior recognition model for public safety early warning, leveraging transfer learning. The model's image features are transferred from ResNet18 to enhance adaptability and reduce training costs, while abnormal behavior features are obtained through training on the UCSD dataset. We provide a detailed introduction to the basic concepts and theoretical foundations of transfer learning, describes the model design and training process, and successfully constructs an abnormal behavior recognition model through experiments with transfer learning and the UCSD dataset. The experimental results demonstrate the model's superior adaptability and accuracy, offering substantial theoretical and practical value in the field of public safety early warning. This study not only fills the gap in cross-domain abnormal behavior recognition technology but also provides a new path for the rapid establishment of highly adaptable abnormal behavior recognition models, showcasing significant application value.
transfer learning, abnormal behavior recognition, public safety early warning, model adaptability
With the acceleration of urbanization and increasing population density, public safety issues have become increasingly prominent. Abnormal behavior recognition, as a key technology in public safety early warning monitoring, can effectively detect and prevent potential safety threats [1]. However, traditional abnormal behavior recognition models typically rely on large amounts of labeled data for training and exhibit poor adaptability across different scenarios, imposing high demands on model training costs and practical applications. Transfer learning, as a method that can utilize existing knowledge to improve the learning efficiency of new tasks, has the potential to reduce model training costs and enhance model adaptability. Therefore, researching an abnormal behavior recognition model based on transfer learning holds significant theoretical and practical significance.
This study aims to rapidly construct an abnormal behavior recognition model suitable for public safety early warning monitoring through transfer learning technology. The specific objectives include: (1) Enhancing model adaptability: By transferring image features from the ResNet18 model, the model can better adapt to different monitoring scenarios; (2) Reducing training costs: Utilizing transfer learning to reduce dependence on large amounts of labeled data, thereby lowering the time and resource consumption for model training; (3) Validating model effectiveness: Through experiments on the UCSD dataset, validating the accuracy and robustness of the model in abnormal behavior recognition.
The main contributions of this research are as follows: (1) Model design: Proposing a cross-domain abnormal behavior recognition model based on transfer learning, and detailing its design ideas and training methods; (2) Experimental validation: Validating the effectiveness of the proposed model through experiments on the UCSD dataset, demonstrating its advantages in adaptability and accuracy; (3) Application prospects: Exploring the application potential of the model in public safety early warning monitoring, and proposing a new path for the rapid establishment of efficient abnormal behavior recognition models based on transfer learning technology. Through this research, the paper not only provides a new technical solution for cross-domain abnormal behavior recognition but also explores new possibilities for the application of transfer learning in the field of public safety. These contributions have significant theoretical and practical implications for enhancing urban safety monitoring capabilities and public safety early warning efficiency.
Abnormal behavior recognition technology has wide applications in fields such as public safety, traffic monitoring, and intelligent surveillance. Traditional methods for abnormal behavior recognition primarily rely on statistical methods and machine learning algorithms, such as Gaussian Mixture Models (GMM) [2-4], Support Vector Machines (SVM) [5, 6], and Hidden Markov Models (HMM) [7, 8]. While these methods have achieved certain successes in specific scenarios, they often require extensive manual feature engineering and do not perform well in complex environments.
In recent years, with the rapid development of deep learning technology, methods based on Convolutional Neural Networks (CNN) and Long Short-Term Memory Networks (LSTM) have gradually become research hotspots for abnormal behavior recognition [9]. Deep learning methods can automatically extract high-dimensional features from images and videos, significantly improving the recognition accuracy and generalization capabilities of models [10]. For example, CNN-based models have made groundbreaking progress in image classification and object detection tasks and have been widely applied to abnormal behavior recognition [11]. However, training deep learning models typically requires a large amount of labeled data, which may be limited by data acquisition and labeling costs in practical applications.
Transfer learning refers to applying a model trained in a source domain to a target domain, thereby improving the learning efficiency of tasks in the target domain [12]. Transfer learning methods can effectively utilize large amounts of data and training experience from the source domain, reducing the need for large amounts of labeled data in the target domain, and have achieved significant results in fields such as computer vision and natural language processing.
In the field of abnormal behavior recognition, cross-domain transfer learning research is gradually gaining attention. Researchers have attempted to apply pre-trained deep learning models to abnormal behavior detection to address issues of data scarcity and high labeling costs. For example, using ResNet models pre-trained on ImageNet for feature extraction, followed by fine-tuning on the target dataset, can significantly enhance the model's recognition performance [13]. Other transfer learning strategies, such as domain adaptation [14], adversarial training [15], and multi-task learning [16], have also been introduced into abnormal behavior recognition to further improve the model's adaptability and robustness.
Despite the broad application prospects of transfer learning in abnormal behavior recognition, several challenges remain. Firstly, differences between the source domain and the target domain may lead to negative transfer effects, affecting the model's recognition performance. Secondly, effectively selecting and adapting features from pre-trained models is a problem that needs to be addressed urgently. Finally, in complex and dynamic real-world application scenarios, combining multiple transfer learning strategies to construct efficient abnormal behavior recognition models still requires in-depth research.
Based on the above research status, we propose a cross-domain abnormal behavior recognition model based on transfer learning, aiming to enhance model adaptability and reduce training costs. By transferring image features from the ResNet18 model and fine-tuning on the UCSD dataset, we verify the effectiveness of this method in abnormal behavior recognition, providing an efficient and feasible technical solution for public safety early warning monitoring.
3.1 Transfer learning
Transfer learning is a technique that leverages knowledge learned in one domain to help solve problems in other related domains [17]. In deep learning, transfer learning typically involves applying a pre-trained model (such as ResNet18 trained on ImageNet) to a new task. By transferring the features or parameters of the pre-trained model, the training time and data requirements for the new task can be significantly reduced.
The core idea of transfer learning is to transfer the model parameters learned in the source domain to the target domain for retraining, thereby improving the learning efficiency and model performance of the target domain task. In practical applications, common transfer learning strategies include feature extraction and fine-tuning [18]. Feature extraction involves utilizing the high-level features extracted by the pre-trained model, while fine-tuning further trains the model parameters on the target task.
3.2 Model architecture
To implement a cross-domain abnormal behavior recognition model based on transfer learning, this study designs an architecture that combines pre-trained models and target dataset features.
The input features of the model mainly include two parts: image features and abnormal behavior features. Image features are extracted by transferring the pre-trained ResNet18 model, leveraging the rich feature information learned by ResNet18 on large-scale datasets. Abnormal behavior features are obtained through training on the UCSD dataset, which includes the following processes: (1) Image feature extraction: Using the pre-trained ResNet18 model, removing the final fully connected layer to extract high-level feature representations of the input images; (2) Abnormal behavior feature extraction: Training on the UCSD dataset to obtain abnormal behavior features for further fine-tuning of the model.
The overall structure of the model includes the following components: (1) Pre-trained model component: Utilizing the pre-trained ResNet18 model to extract high-level features of the input images through its convolutional layers; (2) Feature fusion layer: Combining the image features extracted from ResNet18 with the abnormal behavior features obtained from training on the UCSD dataset to form a comprehensive feature representation; (3) Classification layer: Adding fully connected layers and a Softmax classifier on top of the feature fusion layer for classifying and recognizing abnormal behaviors.
3.3 Training methods and optimization strategies
To ensure efficient training and superior performance of the model, this study adopts the following training methods and optimization strategies:
Data preprocessing. The specific steps include: (1) Image preprocessing: Standardizing the input images and adjusting the image sizes to fit the input requirements of ResNet18; (2) Data augmentation: Using data augmentation techniques such as random cropping, rotation, and flipping to increase the diversity of the training data and prevent overfitting.
Model training and parameter tuning. The specific steps include: (1) Pre-trained model transfer: Transferring the pre-trained ResNet18 model to the abnormal behavior recognition task by freezing some convolutional layer parameters to fine-tune only the fully connected layers; (2) Fine-tuning strategy: Fine-tuning the model on the UCSD dataset by gradually unfreezing the convolutional layers of ResNet18 to adapt to the data distribution of the target task; (3) Hyperparameter optimization: Optimizing the model's learning rate, batch size, and other hyperparameters through grid search or random search methods to enhance model performance.
Transfer learning strategies. The specifics include: (1) Combination of feature extraction and fine-tuning: Initially training with high-level features extracted by ResNet18, followed by fine-tuning to improve model adaptability to the target task; (2) Multi-task learning: Jointly training multiple related tasks using shared features to enhance the model's generalization capability.
4.1 Dataset
This study utilizes the UCSD Anomaly Detection Dataset, a widely used video dataset for anomaly detection, primarily aimed at researching and evaluating anomaly detection algorithms [19]. The dataset includes two scenarios: Ped1 and Ped2, both consisting of video surveillance data in a campus environment. The videos contain normal pedestrian activities and a few abnormal events (such as cycling or driving through walkways). Each scenario provides detailed frame-level and pixel-level annotations for training and evaluating models.
4.2 Experimental setup and evaluation metrics
To evaluate the proposed transfer learning-based abnormal behavior recognition model, the study conducted the following experimental setup: (1) Data division: The UCSD dataset was divided into a training set and a test set, where the training set was used for model training and fine-tuning, and the test set was used to evaluate model performance; (2) Feature extraction: The pre-trained ResNet18 model was used to extract image features, and fine-tuning was conducted on the UCSD dataset; (3) Model training: The Adam optimizer was used for training, with an initial learning rate set to 0.001, which gradually decayed over time. The cross-entropy loss function was used during training.
To comprehensively evaluate the model's performance, the study employed the following evaluation metrics: (1) Accuracy: Represents the proportion of correct predictions made by the model; (2) Precision: Indicates the proportion of true abnormal behaviors among those predicted as abnormal by the model; (3) Recall: Represents the proportion of all correctly identified abnormal behaviors by the model; (4) AUC (Area Under Curve) [20]: The area under the ROC curve, used to assess the model's ability to distinguish between normal and abnormal behaviors.
4.3 Experimental results and analysis
The experimental results on the UCSD dataset show that the transfer learning-based abnormal behavior recognition model performs exceptionally well across various evaluation metrics. Compared to traditional abnormal behavior recognition methods, the transfer learning model significantly improves accuracy, precision, recall, and AUC. The specific results are shown in Table 1.
Table 1. Experimental results
|
Model |
Accuracy |
Precision |
Recall |
AUC |
|
GMM |
85.2% |
82.4% |
80.1% |
0.87 |
|
SVM |
87.6% |
85.7% |
83.3% |
0.89 |
|
Proposed Method |
99.4% |
96.3% |
95.8% |
0.97 |
The experimental results demonstrate that the transfer learning-based model has significant advantages in adaptability and accuracy. By transferring the image features of the ResNet18 model to the abnormal behavior recognition task, the model can effectively utilize the feature representations learned by the pre-trained model on a large-scale dataset, significantly enhancing performance in the target task.
Moreover, the transfer learning model exhibits strong adaptability in different scenarios, maintaining high accuracy and robustness across various environments. This indicates that transfer learning not only reduces model training costs but also enhances the model's generalization ability in practical applications.
Through the above experiments and result analysis, this study verifies the effectiveness and practicality of the transfer learning-based abnormal behavior recognition model in public safety early warning monitoring, providing a solid theoretical and experimental foundation for the rapid construction of efficient abnormal behavior recognition systems.
5.1 Model advantages
The advantages of the proposed model are as follows: (1) Strong adaptability: By applying the pre-trained ResNet18 model to the abnormal behavior recognition task through transfer learning, the model's adaptability in different scenarios is significantly enhanced. This allows the model to maintain efficient abnormal behavior recognition performance in various public safety early warning monitoring environments. (2) Low training costs: The application of transfer learning reduces the dependence on large amounts of labeled data. By leveraging the features of existing pre-trained models, the time and resources required for model training are significantly reduced. This is particularly important in situations where data acquisition and labeling costs are high. (3) High accuracy: Experimental results show that the transfer learning-based abnormal behavior recognition model performs excellently in terms of accuracy, precision, and recall, significantly outperforming traditional abnormal behavior recognition methods. This validates the effectiveness of transfer learning in improving model performance.
5.2 Model limitations
The limitations of the proposed model are as follows: (1) Risk of negative transfer: Although transfer learning can effectively utilize the knowledge of pre-trained models, differences between the source domain and the target domain may lead to negative transfer effects, impacting model performance. Special attention is required when selecting pre-trained models and adjusting transfer strategies. (2) Dependence on pre-trained models: The ResNet18 model used in this study is pre-trained on the ImageNet dataset. While its features are generally applicable, certain specific abnormal behavior recognition tasks may still require further customization and fine-tuning. (3) Scenario dependence: Despite the enhanced adaptability through transfer learning, the model's generalization capability may still be limited in extreme or unconventional scenarios. Specialized optimization and adjustments based on specific application contexts are necessary.
We proposed and implemented a cross-domain abnormal behavior recognition model based on transfer learning, aiming to enhance the performance of abnormal behavior recognition in public safety early warning monitoring. By transferring image features from the pre-trained ResNet18 model and fine-tuning on the UCSD dataset, we successfully constructed an efficient and highly adaptable abnormal behavior recognition system. We provided a detailed introduction to the basic concepts and theoretical foundations of transfer learning, described the model design and training process, and validates the model's effectiveness through experiments.
The value of this study is reflected in the following aspects: (1) Enhanced adaptability: Using transfer learning technology, the model leverages the pre-trained ResNet18 model to extract image features, significantly improving its adaptability and generalization capability across different scenarios; (2) Reduced training costs: Transfer learning reduces the need for large amounts of labeled data, significantly lowering the time and resource costs for model training, making the abnormal behavior recognition model more practical for real-world applications; (3) High accuracy: Experimental results show that the transfer learning-based model excels in terms of accuracy, precision, recall, and AUC, validating its efficiency and reliability.
This study not only provides a new technical solution for cross-domain abnormal behavior recognition but also opens new pathways for the application of transfer learning technology in public safety early warning monitoring. By rapidly constructing efficient abnormal behavior recognition models, the study can enhance the real-time and accuracy of public safety monitoring systems, effectively preventing and responding to potential safety threats.
This work was supported by the Guangdong Key Area R&D Program (Grant No.:2023B1111040001).
[1] Mabrouk, A.B., Zagrouba, E. (2018). Abnormal behavior recognition for intelligent video surveillance systems: A review. Expert Systems with Applications, 91: 480-491. https://doi.org/10.1016/j.eswa.2017.09.029
[2] Ratre, A., Pankajakshan, V. (2018). Tucker tensor decomposition‐based tracking and Gaussian mixture model for anomaly localisation and detection in surveillance videos. IET Computer Vision, 12(6): 933-940. https://doi.org/10.1049/iet-cvi.2017.0469
[3] Piyathilaka, L., Kodagoda, S. (2013). Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features. In 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), Melbourne, VIC, Australia, pp. 567-572. https://doi.org/10.1109/ICIEA.2013.6566433
[4] Luo, Z., He, W., Liwang, M., Huang, L., Zhao, Y., Geng, J. (2017). Real-time detection algorithm of abnormal behavior in crowds based on Gaussian mixture model. In 2017 12th International Conference on Computer Science and Education (ICCSE), Houston, TX, USA, pp. 183-187. https://doi.org/10.1109/ICCSE.2017.8085486
[5] Pan, Y., Zhao, X., Xu, Z., Li, J., Li, Y., Liu, L. (2021). Research on abnormal behavior recognition of buses based on improved support vector machine. Journal of Advanced Transportation, 2021(1): 2020882. https://doi.org/10.1155/2021/2020882
[6] Palaniappan, A., Bhargavi, R., Vaidehi, V. (2012). Abnormal human activity recognition using SVM based approach. In 2012 International Conference on Recent Trends in Information Technology, Chennai, India, pp. 97-102. https://doi.org/10.1109/ICRTIT.2012.6206829
[7] Wang, L., Zhou, Y., Li, R., Ding, L. (2022). A fusion of a deep neural network and a hidden Markov model to recognize the multiclass abnormal behavior of elderly people. Knowledge-Based Systems, 252: 109351. https://doi.org/10.1016/j.knosys.2022.109351
[8] Yin, J., Meng, Y. (2009). Abnormal behavior recognition using self-adaptive hidden Markov models. In: Kamel, M., Campilho, A. (eds) Image Analysis and Recognition (ICIAR 2009). Lecture Notes in Computer Science, vol 5627. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02611-9_34
[9] Jia, S., Hui, F., Li, S., Zhao, X., Khattak, A.J. (2020). Long short‐term memory and convolutional neural network for abnormal driving behaviour recognition. IET Intelligent Transport Systems, 14(5): 306-312. https://doi.org/10.1049/iet-its.2019.0200
[10] Wang, J., Xia, L. (2019). Abnormal behavior detection in videos using deep learning. Cluster Computing, 22(Suppl 4): 9229-9239. https://doi.org/10.1007/s10586-018-2114-2
[11] Ganga, B., Lata, B.T., Venugopal, K.R. (2024). Object detection and crowd analysis using deep learning techniques: Comprehensive review and future directions. Neurocomputing, 597: 127932. https://doi.org/10.1016/j.neucom.2024.127932
[12] Weiss, K., Khoshgoftaar, T.M., Wang, D. (2016). A survey of transfer learning. Journal of Big Data, 3: 1-40. https://doi.org/10.1186/s40537-016-0043-6
[13] Abosaq, H.A., Ramzan, M., Althobiani, F., Abid, A., Aamir, K.M., Abdushkour, H., Irfan, M., Gommosani, M.E., Ghonaim, S.M., Shamji, V.R., Rahman, S. (2023). Unusual driver behavior detection in videos using deep learning models. Sensors, 23(1): 311. https://doi.org/10.3390/s23010311
[14] Ngiam, J., Peng, D., Vasudevan, V., Kornblith, S., Le, Q.V., Pang, R. (2018). Domain adaptive transfer learning with specialist models. arXiv:1811.07056. https://doi.org/10.48550/arXiv.1811.07056
[15] Deng, Z., Zhang, L., Vodrahalli, K., Kawaguchi, K., Zou, J.Y. (2021). Adversarial training helps transfer learning via better representations. Advances in Neural Information Processing Systems, 34: 25179-25191.
[16] Zhang, Y., Ying, S., Wen, Z. (2022). Multitask transfer learning with kernel representation. Neural Computing and Applications, 34(15): 12709-12721. https://doi.org/10.1007/s00521-022-07126-3
[17] Liu, R., Shi, Y., Ji, C., Jia, M. (2019). A survey of sentiment analysis based on transfer learning. IEEE Access, 7: 85401-85412. https://doi.org/10.1109/ACCESS.2019.2925059
[18] Feng, L., Yang, Y., Tan, M., Zeng, T., Li, Z., Tang, H., Li, Z., Niu, Z., Feng, F. (2023). Adaptive multi-source domain collaborative fine-tuning for transfer learning. PeerJ Computer Science, 10: e2107. https://doi.org/10.7717/peerj-cs.2107
[19] Lalit, R., Purwar, R.K., Verma, S., Jain, A. (2022). Crowd abnormality detection in video sequences using supervised convolutional neural network. Multimedia Tools and Applications, 81(4): 5259-5277. https://doi.org/10.1007/s11042-021-11781-4
[20] Huang, J., Li, J., Wu, W. (2022). A new method of pedestrian abnormal behavior detection based on attention guidance. Advances in Multimedia, 2022(1): 1038225. https://doi.org/10.1155/2022/103822