A Multimodal Deep Learning and Internet of Things Framework for Real-Time Stray Dog Behavior Detection and Child Safety Monitoring

Recent advancements in computer vision, DL, and multimodal AI systems have significantly accelerated research in automated animal behavior monitoring and activity recognition. Traditional dog activity recognition systems initially relied on image-based classification models that focused on identifying simple postures and actions from static frames. Nolasco et al. [10] proposed a DL framework for dog activity recognition using the InceptionV3 CNN architecture deployed on a Raspberry Pi platform for edge-based monitoring applications. Their system classified dog activities such as sitting, standing, and lying using captured visual data and achieved approximately 88% classification accuracy. The major contribution of this work was demonstrating that lightweight CNN architectures could be implemented in low-cost embedded systems for pet monitoring. However, the model primarily depended on static image classification and failed to capture temporal movement patterns such as continuous walking, running, or transitions between activities, which are critical for real-world behavioral monitoring.

To address limitations in vision-only systems, researchers explored wearable sensor-based activity recognition approaches. Syeda et al. [11] developed a dog activity detection and recognition framework that utilized wearable accelerometer and motion sensors to capture movement patterns of dogs. ML algorithms were applied to classify activities such as running, walking, sitting, standing, and resting. The study demonstrated promising performance with nearly 79% recognition accuracy, showing that motion-based data can effectively capture behavioral information. However, wearable-based systems introduce practical challenges such as hardware cost, battery dependency, discomfort for animals, and reduced scalability when deployed for large populations of pets or stray dogs.

Recent studies have increasingly focused on multimodal learning approaches, where multiple data sources are combined to improve recognition performance. Kim and Moon [12] introduced a multimodal dog behavior recognition framework that combined camera-based visual information with wearable sensor data. Their approach first utilized YOLOv3, YOLOv4, and Faster R-CNN models to detect dogs in video streams and then integrated extracted visual features with sensor-generated motion signals using a CNN-Long Short-Term Memory (LSTM) fusion architecture. The CNN component extracted spatial features from images, while the LSTM network learned temporal dependencies from sequential behavioral data. This hybrid approach significantly improved activity classification performance compared to single-modal systems. However, the dependency on multiple sensors and multimodal synchronization increased system complexity and reduced deployment flexibility in real-time environments.

Similarly, Garcia-Loya et al. [13] expanded multimodal animal monitoring by introducing an automatic canine emotion recognition framework that combined visual information, inertial sensor readings, and physiological signals. Their system classified complex emotional states such as frustration, playfulness, abandonment, and petting responses. ML algorithms including Support Vector Machines (SVM), k-Nearest Neighbor (kNN), and Extra Trees classifiers were used for classification. The framework achieved an impressive F1-score of 0.96, demonstrating the effectiveness of combining heterogeneous data sources. Despite high accuracy, the requirement for physiological sensors and extensive preprocessing pipelines made the approach computationally expensive and difficult to deploy in real-world surveillance applications.

Real-time object detection techniques have also contributed significantly to intelligent monitoring applications. He et al. [14] proposed an automatic real-time infant drowning detection system using YOLOv5 and Faster R-CNN models integrated with video surveillance systems. The framework continuously analyzed swimming pool video feeds to detect drowning incidents in real time. YOLOv5 provided faster detection speed, while Faster R-CNN offered improved localization accuracy. Their research demonstrated that DL-based object detection frameworks can be highly effective in safety-critical real-time surveillance tasks. Although this work focused on human safety rather than animal behavior, its real-time video monitoring methodology provides valuable insights for developing dog activity detection systems using surveillance cameras.

In addition to detection accuracy, researchers have also addressed the issue of model robustness against adversarial threats. Li et al. [15] proposed DOG: An Object Detection Adversarial Attack Method, which revealed significant vulnerabilities in modern object detection systems. Their framework generated adversarial perturbations capable of misleading DL-based detectors such as YOLO models. The study demonstrated that even highly accurate detection systems may fail under adversarial conditions, emphasizing the importance of developing robust and secure detection frameworks for real-world deployment, particularly in surveillance applications.

Beyond vision-based systems, multimodal intelligence research has expanded into other domains such as audio and contextual data analysis. Dsouza et al. [16] proposed a multi-modal comparative analysis on audio dub detection using AI, where multiple audio features and DL models were used to identify manipulated speech content. Their study showed that multimodal feature extraction significantly improves classification reliability. Similarly, Rahman et al. [17] introduced CAMFusion, a context-aware multimodal fusion framework that integrated video and textual cues for sarcasm and humor detection. Their model effectively fused heterogeneous data streams to improve contextual understanding and classification accuracy. Although these studies belong to different application domains, they strongly support the effectiveness of multimodal feature fusion techniques for improving detection reliability in complex real-world environments.

Several studies have also focused on automated animal monitoring using surveillance-based systems for public safety. Recent stray dog detection frameworks employed YOLOv5-based object detection models to identify stray dogs in urban environments through CCTV surveillance. These systems enabled automated tracking and alert generation for municipal safety applications. While effective for localization and tracking, such systems primarily focus on detecting the presence of animals rather than understanding specific activities or behavioral states.

Overall, previous research demonstrates significant progress in CNN-based classification, sensor-driven recognition, multimodal fusion, and real-time object detection. However, important limitations still remain. Static CNN models fail to capture temporal behavioral information, wearable systems reduce practicality, multimodal systems often increase computational overhead, and object detection models primarily focus on localization rather than activity recognition. Furthermore, adversarial vulnerabilities can compromise system reliability in real-world environments. Motivated by these research gaps, the proposed work aims to develop an efficient dog activity detection and recognition system that combines robust object detection with accurate activity classification while maintaining real-time performance and reducing dependency on expensive hardware or complex multimodal infrastructures.

For reliable detection of aggressive dog behavior, the proposed methodology presents an end-to-end intelligent monitoring system that integrates audio signal processing, DL-based sound classification, computer vision-based object detection, IoT hardware integration, and real-time alert communication. The system processes both audio and visual inputs to distinguish dog behavior. The proposed framework begins by collecting dog vocal sounds such as bark, growl, and grunt using a sound sensor, while real-time images and video frames are captured using a camera module. Additionally, the ultrasonic sensor is utilized to estimate the distance between the detected dog and nearby individuals, enabling proximity-based risk assessment. The emergency push button provides manual alert activation during critical situations. The collected multimodal data is then passed through preprocessing and classification stages for behavior detection.

The audio preprocessing stage uses Librosa to convert raw sound signals into meaningful frequency-domain features. Since raw audio signals contain noise and unstructured waveform information, feature extraction is required before classification. Mel Frequency Cepstral Coefficients (MFCCs) are extracted from the audio signal and mathematically represented as:

$\operatorname{MFCC}(n)=\sum_{k=1}^K \log \left(S_k\right) \cos \left[\frac{\pi n(k-0.5)}{K}\right]$          (1)

Eq. (1) transforms raw dog vocal signals into compact spectral representations by extracting important frequency features. Here, $S_k$ represents Mel-scaled spectral coefficients, $K$ represents the total number of Mel filters, and $n$ denotes the coefficient index. This feature extraction process helps the system differentiate between bark, growl, and grunt sounds based on their acoustic patterns. Growling sounds are treated as strong indicators of aggressive behavior.

After feature extraction, the generated MFCC feature maps are passed to a CNN for audio classification [19]. CNN extracts high-level patterns from audio spectrograms through convolution operations, which are mathematically represented as:

$F(x, y)=\sum_{i=0}^m \sum_{j=0}^n I(x-i, y-j) W(i, j)+b$               (2)

Eq. (2) represents the convolution operation performed in each CNN layer, where I denotes the input feature map, W represents the convolution kernel, and b is the bias term. This operation helps detect important sound features such as frequency peaks, tonal variations, and aggression-related vocal signatures. The extracted features are passed through the Rectified Linear Unit activation function to introduce non-linearity:

$ReLU(x)=\max (0, x)$        (3)

Eq. (3) removes negative values and improves training efficiency by enabling faster convergence. Finally, the CNN uses a SoftMax activation function in the output layer to classify the audio signals into bark, growl, and grunt categories:

$P\left(y_i\right)=\frac{e^{z_i}}{\sum_{j=1}^n e^{z_j}}$         (4)

Eq. (4) converts the network output into probability scores for each sound class. The highest probability class is selected as the final prediction.

Simultaneously, the visual aggression detection module processes images captured from the camera. The captured images are resized to a fixed dimension of 640 × 640 pixels and normalized before being passed to the YOLOv5 model. Image normalization is mathematically represented as:

$X_{\text {norm }}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$           (5)

Eq. (5) ensures that pixel values are scaled between 0 and 1, which improves training stability and detection performance. Data augmentation techniques such as rotation, brightness adjustment, scaling, and horizontal flipping are also applied to improve model robustness under varying environmental conditions.

The YOLOv5 model is used for real-time dog detection and aggression classification. YOLO predicts bounding box coordinates along with object confidence scores and class labels. The bounding box prediction is mathematically represented as:

$B=(x, y, w, h, c)$        (6)

Eq. (6) where x and y represent the center coordinates of the bounding box, w and h represent width and height, and c represents the confidence score. This enables accurate localization of dogs in real-time video streams.

The performance of object detection is measured using Intersection over Union (IoU), which evaluates overlap between predicted and actual bounding boxes:

$I o U=\frac{\text { Area of Overlap }}{\text { Area of Union}}$          (7)

Eq. (7) ensures accurate localization by comparing predicted bounding boxes with ground truth annotations. Higher IoU values indicate better detection performance.

The final stage of the system combines both audio and visual outputs through multimodal decision fusion [20]. The detection logic is represented as:

$Aggression = Audio_{\text {growl }} \cap Visual $         (8)

Eq. (8) confirms aggressive behavior only when both audio and visual indicators suggest aggression. For example, if the audio model detects growling and the YOLOv5 model detects angry behavior, the system triggers an alert. This multimodal fusion significantly reduces false positives compared to single-modal systems.

To train the CNN model, categorical cross-entropy loss is used since the audio classification problem involves multiple classes. The loss function is represented as:

$L=-\sum_{i=1}^N y_i \log \left(\widehat{y}_l\right)$           (9)

Eq. (9) measures the difference between predicted probabilities and actual class labels, enabling the CNN model to optimize classification performance.

The identified emotional states are mapped to risk levels. High Risk Aggressive behavior such as growling or continuous barking and angry behavior. Low Risk: Happy and Relaxed states Medium Risk: Sad behavior (proximity) This decision-making process improves the predictability of encounter prevention.

Once behavior is confirmed, the ESP32 microcontroller receives the detection signal and activates the alert system. The LCD displays warning message and Telegram API sends real-time alerts to users. This integrated methodology ensures automated, interpretable, scalable, and real-time dog behavior detection for residential areas and public safety applications.

The proposed system is to be used in public places such as schools, parks, playgrounds, and residential communities. The ESP32-based monitoring unit constantly monitors the environment via camera and audio sensors. The ESP32 activates local warning mechanisms through a buzzer that is present in a wrist band and while simultaneously sending Telegram notifications to guardians or authorities. The wrist band currently shown in this work can be worn by the child in real time or some neck piece can be developed as a product in the future based on the requirement and convenient.

The experimental results demonstrate that the proposed framework achieved strong classification performance across all four emotional classes. The YOLOv5 model effectively detects dogs in images and video frames and extracts behavioral patterns associated with emotional states. Simultaneously, the audio analysis module uses Librosa to extract acoustic features such as MFCCs, spectral centroid, chroma features, and zero-crossing rate, which help identify emotional variations from barking sounds and vocal patterns. The combination of visual and audio modalities improves classification reliability compared to single-modal systems.

The real-time alert mechanism enhances the practical usability of the proposed system. When the system detects aggressive or abnormal emotional states such as Angry or prolonged Sad behavior, alerts are generated and sent to pet owners or monitoring systems. This feature makes the framework suitable for smart pet care systems, veterinary monitoring, and animal welfare applications. Despite its strong performance, the system has certain limitations. The primary limitation is dataset size. The available dataset contains a limited number of samples across all four emotional classes, which may restrict model generalization for different dog breeds and environmental variations. Although augmentation techniques were applied, larger datasets would improve robustness.

Another limitation involves noisy environments during audio analysis. Background sounds such as human voices, traffic noise, or other animal sounds may interfere with Librosa feature extraction and reduce classification accuracy. Similarly, visual detection performance may decline under poor lighting conditions, occlusions, motion blur, or multiple dogs appearing in the same frame. Additionally, emotional behavior in dogs can be highly complex and sometimes overlapping. For example, playful barking may occasionally resemble aggressive barking, which can create classification ambiguity. The current system also processes audio and video streams separately before combining outputs, which may introduce slight delays in real-time scenarios.

Future work can improve the framework by integrating advanced multimodal DL architectures such as CNN-LSTM, Transformers, or attention mechanisms for better fusion of visual and audio features. Expanding datasets, improving real-time optimization, and integrating wearable IoT sensors can further enhance system reliability.

The proposed framework has practical applications in:

•Smart pet monitoring systems

•Veterinary healthcare

•Animal welfare monitoring

•Pet daycare centers

•Security monitoring systems

•Behavioral research laboratories

Although the model performs effectively, challenges such as limited dataset availability, environmental noise, and overlapping emotional behaviors still exist. Future research will focus on larger multimodal datasets, improved DL architectures, IoT integration, and enhanced real-time deployment.

The performance of the proposed framework may decrease under challenging conditions such as low-light environments, severe occlusions, long-distance detection, and crowded scenes containing multiple stray dogs. Future work will focus on improving robustness under these real-world conditions through larger datasets and advanced vision models.

Overall, this work contributes a scalable, intelligent, and real-time solution for automated dog emotion recognition and provides a strong foundation for future advancements in animal behavior analysis systems.