Normalized Attention Neural Network with Adaptive Feature Recalibration for Detecting the Unusual Activities Using Video Surveillance Camera

Surveillance systems based on video cameras are a cost-effective and efficient way to keep goods, property, and people safe. Video cameras are employed for long-distance, weather-independent observation of dynamic situations where light, temperature, and visibility are all subject to continual change. There is a global proliferation of public video surveillance systems, which may yield useful and detailed data for a wide range of safety-related tasks [1]. Crime and violence prevention rely heavily on video monitoring, but the necessity of studying hours of material diminishes the opportunity to make quick choices [2]. Several research have been published on the topic of automatically detecting violent incidents in films in an effort to relieve authorities of the load of having to view hours of footage in order to identify occurrences that only last seconds. Recent publications [3-5] emphasised the accuracy of systems in violence discovery, while earlier efforts [6-8] relied on custom-built features and flow forms are hallmarks of outdated approaches of identifying actions. Extraction of spatial-temporal features from films, or features that capture both the spatial information in a single frame and the gesture info in a series of edges, has been demonstrated to be possible with the use of deep learning approaches [9].

Surveillance in a "smart city" may be used for a wide variety of tasks, including as keeping tabs on city traffic [10], identifying structural damage to buildings [11], identifying instances of violence [12], and managing the aftermath of natural catastrophes [13]. The sheer volume of video feeds might potentially overwhelm human operators. For this reason, much work was put into discovering means of automatically processing such data for the purposes of monitoring for anomalous activity and discarding securely unneeded data. In the context of smart city monitoring, the ability to identify acts of violence is a key concern. With a wireless sensor network infrastructure, we can solve the problem at a reasonable cost. It's important to note that such a solution entail in contrast to current methods, the work here presents a distributed algorithm that can operate on sensor nodes and identify violent behaviour based on camera input while also reducing the communication load on nodes with limited CPU resources and bandwidth.

In contrast to current methods, the work presented here provides a distributed, sensor-level algorithm that can identify violent behaviour from camera input while simultaneously reducing communication overhead by simply transmitting alerts. The most advanced algorithms for this task rely heavily on computer vision techniques including segmentation, tracking, and action identification, all of which need a lot of processing power [14]. The study of identifying violent acts is a special example of human action recognition [15]. Recognizing an action takes considerable effort due to factors including occlusion, size, and busy backgrounds. Furthermore, there are unique challenges associated with violence detection, the first of which is establishing a workable definition of violence. Humans have a hard difficulty differentiating between violent and nonviolent scenarios. Certain physical and social behaviours, such as jogging, dancing, and interacting with other people, might superficially resemble aggressive ones.

Using a supervised learning framework, this research suggests recasting. Our definition of an anomalous occurrence determines whether or not we should be concerned about a wide variety of odd behaviours in the actual world. However, in this paper, we emphasis on the UCF-Crime dataset, which contains a great deal of aberrant, unlawful, and violent behaviour caught on public surveillance cameras and which can cause serious issues for both individuals and the population as a whole. For feature extraction in our suggested model, we turned to MRCNN with AFR. Next, we include the video dataset into the model architecture by including an NAN, that is well-suited to this type of data. After that, the model reports back on whether or not the given video contains any criminal activity. This approach has the potential to boost the efficiency with which humans can detect permanent harm and reduce associated costs. In this research work, starting with an MRCNN for feature extraction and AFR for fine-tuning, this architecture has a number of key components (AFR). To increase the quality of the features extracted by the MRCNN, the AFR replicas the inter-dependencies among the features to enhance the quality of the low- and high-frequency features extracted. Then, a normalized attention network (NAN) is used to learn the relationships between channels, which used to identify the violence and speeds up the convergence process for training a perfect. Furthermore, the dataset took real-time security camera feeds from a variety of subjects and situations, as opposed to the hand-crafted datasets utilized in prior efforts. We also demonstrate the method's capability of assigning the correct category to each anomaly by classifying normal and abnormal occurrences. The method divided the information gathered into three primary groups: those in need of fire protection, those experiencing theft or violence, and everyone else. The study applied the proposed approach to the UCF-Crime dataset, where it outperformed other models on the same dataset. The following is a brief summary of the method's key contribution.

·MRCNN-ARF and NAN are used together to identify anomalies in video surveillance.

·For this study, we employed the UCF-Crime dataset, which is comprised of 13 types of aberrant events depicted in natural settings captured by security cameras.

·We established two by separating the typical scenarios from the out-of-the-ordinary ones in order to gain a deeper comprehension of each anomaly type.

Section 2 of this paper discusses how other studies in the field of anomaly detection in security cameras employ a variety of models, each of which is a sub-model of the overall concept. After that, in Section 3, we detail our suggested model. Section 4 presents to critically assess our work via a series of experiments. Section 5 discusses the conclusion and future scope.

3.1 Dataset

In this study, we apply the suggested model to the UCF-Crime dataset [23], which contains a large amount of footage from public surveillance cameras documenting anomalous, unlawful, and violent behaviour in settings as diverse as schools, businesses, and streets. This dataset was chosen because its events are representative of those that occur often and in a variety of settings [24-26]. Furthermore, these aberrant behaviours can cause significant issues for both people and society. Several articles depend on artificially constructed datasets or datasets that have a common context and history (such as hockey battle datasets and movie datasets), neither of which is representative of real-world experience. This dataset contains extensive, unedited footage from 13 different types of anomalous occurrences and one type of normal event [27]. In Figure 1, sample images of UCF-Crime dataset is depicted. In our trials, we alienated the data into a training set of 75% and a testing set of 25% so that we could fairly compare our results to those of other researchers in the area.

1.png

Figure 1. Examples of UCF-Crime dataset

We used the UCF-Crime dataset in its four different iterations: UCF-Crimes, Binary, 4MajorCat, and NREF. Our starting point is a 14-class dataset called UCF-Crimes. Within the confines of the Binary, we group together the 13 outlier occurrences into a single category. We divided the usual occurrences into one category and the anomalous ones into three major ones for the 4MajorCat. Theft, Vandalism, and Violence are the three categories that encompass these actions. Another example of skewed data, the NREF isolates just three outliers.

Table 1. Sum of videos for Dual and Ucfcrimes datasets

Table 2. Sum of videos for 4MajCat and NREF datasets

In this dataset, aberrant films have portions that feature abnormalities judged abnormal, rather than utilising predetermined abnormal and normal videos. On the other hand, the rest of the video is classified as neutral. Also, we condensed all NREF videos into 10-second clips so that we could extract the most relevant images from each source. Table 1 shows the distribution of video counts across categories in the Ucfcrimes and Binary datasets. In addition, the total sum of videos in the 4MajCat and NREF databases is listed in Table 2.

3.2 Pre-processing

First, we extract individual frames from each video. By counting how many frames in a video file should be ignored, we may determine how many frames can be used to create a subset of n frames. Therefore, if a video file is 60 seconds long, and the video format is set to 30 frames per second, then the total sum of video frames is m=1800. Let's pretend for a moment that n=30 and that, from a total of 1800, you've had to pick only 30. Therefore, once 60 frames have been skipped, we need to pick one. In order to account for spatial mobility in each input, we first selected the frames and then computed the difference between each pair of frames. One set of datasets was pre-processed differently from the others. From the UCF-Crime dataset, we chose to focus on three subsets. We identified the out-of-the-ordinary moment in each video and marked it as a "Anomaly," while the other footage was classified as "Normal." Next, cut all of the videos into equal-sized pieces. Thus, n frames will be chosen from m frames, as was the case before, but now the sum of m is also predetermined. So, in the prior work's agreement, we simply paid attention to the timestamps at which unexpected occurrences occur. In addition, both the anomalous event and the normal set come from the same data source. This implies that in contrast to the initial UCF- dataset, the only difference between the two is the acting, therefore the algorithm is better able to spot out-of-the-ordinary occurrences.

3.3 Feature extraction

In Figure 2, we can see the MRCNN and AFR modules used to extract topographies from the pre-processed frames.

2.png

Figure 2. The MRCNN and AFR components for feature extraction

1) Multi-Resolution CNN: In Figure 2, we can see that we've implemented a multi-resolution CNN architecture to facilitate the extraction of a wide variety of features. Specifically, we use two forks of convolutional layers with varying kernel sizes, with the goal of probing various frequency regions based on the sampling rate of the frames. This is motivated by prior efforts that employed several CNN kernel dimensions to extract low- and high-frequency features. It is becoming more crucial to address multiple frequency bands in order to improve the retrieved characteristics, as different phases are characterised by diverse ranges of frequencies. This is why we employ a variety of kernel sizes to imprisonment a wide variety of timesteps and, by extension, characteristics from a wide variety of frames. First, the 400-kernel wide collects 4-second timesteps, allowing for the collection of a complete cycle of a sinusoidal signal at frequencies as low as 0.25 Hz (T=1/F). In this region, you'll find the delta band. Second, the data corresponds to the alpha and theta bands since each convolution window catches 50 samples (0.5 second) with a kernel of 50.

As can be seen in Figure 2, each of the four branches is made up of three max-pooling layers, with the convolution layers employing a batch normalisation layer and a Gaussian Error Linear function. More specifically, the configuration shown in Figure 2 as Conv1D (64, 50, 6) employs a 1D convolution layer with 64 filters, a 50-by-50-pixel kernel, and a 6-pixel stride. In a similar vein, a maxpooling layer with a kernel size of 8 and a stride of 2 is denoted by the notation MaxPooling (8, 2). Overfitting is mitigated by introducing a dropout step following the initial maxpooling in each of the two branches and again following the concatenation of the two branches, as illustrated in Figure 2.

2) Adaptive Feature Recalibration (AFR): To that end, AFR seeks to adjust the MRCNN-learned features for the purpose of enhancing their performance. Through a residual squeeze and excitation block, the AFR replicas the relationships between the features and adaptively chooses the most discriminative ones. The SE block provides a context-aware technique that aids the network's lower levels in making better use of contextual information from beyond their immediate receptive field. We use two Conv1D (30,1,1) convolutions in the residual SE block, setting both the kernel and stride size to 1 and the activation function to ReLU. The MRCNN-learned feature map I R(Ld) is convolved with itself twice, as shown by F=Conv2(Conv1(I)), to get F.where $F=\{F 1, \ldots, F N\} \in R^{N \times d}$, N is the total sum of features, d is the distance of F_i(1≤i≤N), and Conv1 and Conv2 are procedures in AFR module.

Then, we use a technique called adaptive average pooling to reduce the size of the world's spatial data. $F \in R^{N \times d}$  to s={s₁, ..., s_N}, where si is the mean value of the d data points. Two FC layers are then functional to the assembled data for utilisation. Specifically, after the first layer, a ReLU activation function is used to do dimensionality reduction (as mentioned in Equation 1), and after that, layers are utilised to utilise the gathered information. In Eq. (1), for instance, we see that the first layer is shadowed by a ReLU activation function for dimensionality reduction, while the second layer is shadowed by a smoothing sigmoid activation function for dimensionality cumulative.

$e=\sigma\left(W_2\left(\delta\left(W_1(s)\right)\right)\right) \in R^{N \times d}$               (1)

where, σ and δ mention to sigmoid and ReLU activation purposes correspondingly, and W₁ and W₂ signify the two FC layers in AFR. Then, the feature by e as shadows:

$O=F \otimes e \in R^{N \times d}$            (2)

where, $\otimes$ means that F was multiplied by e on a point-by-point basis. The upgraded chosen characteristics learnt in the residual SE block are likewise merged with the initial input I via a new, streamlined link. The AFR module's end product is:

$X=I+O \in R^{N \times d}$               (3)

Please note that the GELU activation function is used in the MRCNN module since it permits certain negative weights of the input. The subsequent AFR module might be affected by these negative weights and make different choices as a result. Given that ReLU converts all negative weights to zero, the AFR module won't be able to use them, GELU is expected to perform better. In the AFR module, however, we employ ReLU to ensure that the gradient does not explode or vanish, as well as to speed up the calculations and make them more reliably converge. While other activation functions, such as Leaky-ReLU and PReLU, also return negative values, GELU may be preferable due to its more flexible implementation. This is because strong negative activations might have an undesired effect on the total of activations feeding the subsequent layers when using these activation functions. However, GELU differs in that it demonstrates more control to limit the impact of these detrimental activations. By using these extracted features, violence or non-violence are detected by using the proposed classifier.

3.4 Classification

To better understand the connection between channels, we propose a normalised attention network that incorporates 1D normalisation techniques into the already-powerful SENet. We also present the normalised attention network for CNN denoising, wherein each channel is given a certain gain and channels have distinct functions in the subsequent convolution. Assuming that the batch size is N, we will create N random values from the noise level range to use as noise standard deviations during training.

The output Y_R is the quantity of the input X_R of features and output Z:

$Y_R=X_R+Z$                 (4)

Eq. (4) can be reformed as Z=Y_R-X_R, The effort required to learn Z is referred to as remaining learning. The computational cost of training can be reduced by avoiding the vanishing gradient problem and by the use of learned residues. In the following, we'll discuss NAN Block's structure in further depth.

3.4.1 NAN

In Figure 3, we can see the NAN block's structure. Squeeze uses an average pooling function to transform each channel in the input X_N into a point, assuming that there are 96 channels in the X_N. The following is a mathematical formulation of the squeeze operation for each channel $X_N^l$:

$X_N^l=\frac{1}{H \times W} \sum_{i=1}^H \sum_{j=1}^W X_N^l(i, j)$                (5)

where, $X_N^l(i, j)$ is the amplitude at position $(i, j), x_N^l$ is the typical $X_N^l$ value in a channel. 96 neurons make up the fully linked FC layer. Convergence may be sped up and the vanishing gradient problem solved with $1 \mathrm{DBN}$ and Relu layers. If batchsize is set to $\mathrm{k}$, then $\mathrm{k}$ samples will be used in each training procedure. In this section, we will discuss the $1 \mathrm{DBN}$.:

$\tilde{x}_{b n}=\frac{x_{b n}-E\left(x_{b n}\right)}{\sqrt{\operatorname{var}\left(x_{b n}\right)}}$            (6)

$y_{b n}=\gamma \tilde{x}_{b n}+\beta$              (7)

where, x_bn and y_bn are back propagation variables that are changed to reflect changes in the input and output vectors of the BN block. To convert the input to the interval [0,1], the sigmoid function is used on the last layer. Sigmoid function output is denoted by the letter s and is a 96-dimensional vector. as (s₁, s₂, ···, s_n-1, s_n), n is 96. For each channel, the scale process is labeled as shadows:

$Y_N^m=X_N^m . s_m, \quad m=1,2, \ldots, n$            (8)

where, $Y_N^m$ and X^m are the mth stations of Y_N and X_N, respectively. s_m is the mth element of s. From Eq. (8), we find every channel/frame in XN is increased by the conforming element of s, which is used to identify the frame is violence or non-violence.