Statistical Evaluation of Video Summarization Models from an Empirical Perspective

$x_t=\frac{1}{L_W-1} * \sum x$                                  (3)

$E_r=\frac{1}{\left|x_t-x_t^{\prime}\right|}$                        (4)

$R=\frac{\sum E_d}{\sum E_r}$                                               (5)

$E(t)=\alpha * E_d(t)+\partial * R * E_r(t)$                               (6)

where, L_W is the length of the sliding window, x is the number of pixels in the frame, it is the average number of pixels in the sliding window, E_d and E_r are the ratio and division values for energy, and E(t) is the energy for the frames in question. A frameset is retrieved and utilized for summarization if and only if it has a high energy level. Because of this oversimplified assessment, the approach achieves a middling 75% accuracy compared to deep learning models.

Textual descriptions can be used by summarization models to locate crucial scenes. This is a possibility for videos that include annotated texts or have a person speaking directly into the camera. Nonetheless, Otani et al. [3] provides a methodology to extract video summaries from such movies, which might be valuable for certain applications like teaching, product evaluations video analysis, etc. despite their relative scarcity in comparison to the vast array of multimedia data available online. To locate 'entity mentions,' the proposed model employs noun extraction from the text and NLP. To pinpoint shifts in time, we compare these noun entities to what appears in the film. Summary frames, which are frames from a video that have been extracted because of their significant changes, are then organized for entity-based analysis. Figure 3 displays the process's overall flow diagram, with entity analysis and video analysis presented in their respective panels. Because of this association, the model achieves an accuracy of 83%; replacing the video segmentation process with deep learning models might further increase this accuracy. Recurrent neural networks (RNNs), memory networks (MNs), convolutional long short-term memory (LSTM) networks (CNNs), gated recurrent units (GRUs), capsule networks (CNs), generative adversarial networks (GANs), etc. are just some of the methods described in Apostolidis et al. [4], which provides a summary of these models.

3.png

Figure 3. Text-based analysis of videos for summarization [3]

Comparing the performance of these networks with that of other summarization models reveals that CNN-based architectures and Memory Augmented Neural Networks (MANN) get superior results (97% and 94%, respectively). Using a rank-based method, as suggested in Srinivas et al. [5], can further boost these models' efficiency. Here, the ultimate score for each video frame is calculated by combining scores from many processes, such as quality, static attention, temporal attention, and representativeness. The frames are given a score, and those with the highest scores are flagged as "key-frames" and taken out for summary. To get a final video summary, this model employs a post-processing phase in which duplicate key-frames are deleted from the output. This rank-based method only achieves an accuracy of around 79%, which is unacceptable and has to be enhanced with deep learning models. Atencio et al. [6] is an example of a deep learning model that makes advantage of both visual and category variation. Using a deep convolutional network and a pre-trained word-embedding matrix, the authors of this study offer a method for extracting images that are highly distinct from one another. Redundant frames are filtered out by running them via a thresholding mechanism. Figure 4 shows the model's architecture, which relies on the combination of deep features and means activation functions to generate the final video summary. Achieving accuracies between 80% and 90%, this approach is extremely reliant on the existence of word content in the video. In Zhou et al. [7], we see another approach for character-based summarization that relies on the video's textual contents, this one employing entity identification to assess the temporal locations of these entities. These timestamps, along with the user's query, allow the video summarising process to produce only the frames that pertain to the specified entities. For entity detection in input movies, it combines a character-level LSTM model, a neural topic model, and a skip-gram model. Accurate video summarization, measured across several datasets, is possible thanks to the mapping of these items to their associated frames.

4.png

Figure 4. Word embedding extraction from video sequences to obtain final summarization [6]

Algorithms are assessed based on their performance on several datasets, and the correctness of each dataset is measured. For example, the accuracy of LSTM + Random Forest is 70%, the accuracy of the Kronecker-Product Matching Model is 81%, the accuracy of the Multi-level Factorisation Net is 97%, the accuracy of Mancs is 61%, the accuracy of Textual-based Methods is between 70% and 80%, and the accuracy of Deep-Semantic Models is between 87% and 98% for both animation and real-time datasets. Also, deep learning models may be used to enhance the classification performance of application-specific video summarization algorithms. Coarse-grained and fine-grained refinement models based on versions of CNN are proposed for high accuracy [8]. Figure 5 depicts the model's architecture, which shows how the final summary generation process is formed from the coarse refining and fine refining models. Based on the specifics of the dataset, the suggested model may reach an accuracy of between 79% and 92%. Because of its excellent accuracy, the method may be used for any type of video summary. Using the same dataset, we find that AlexNet has 35% accuracy, GoogLeNet has 39% accuracy, SqueezeNet has 79% accuracy, and MobileNet V2 has 45% accuracy, all of which improve the proposed model's real-time deploy-ability. Hybridizing the models' designs to decrease feedback error increases their effectiveness. To improve summarization precision, Ji et al. [9] propose a hybrid model that blends GoogLeNet architecture with several BiLSTM models. To minimize the effects of regression and distribution error in the input video, this model employs a self-attention architecture that gives both additive and multiplicative attention. The following formula represents this mistake (E):

$\begin{aligned} & E_{k l}={ }_n * \sum \mathrm{W}{ }_{i=1}^N \underline{\operatorname{softmax}}\left(Y_i\right) * \log \left[\operatorname{softmax}\left(Y_i\right)\right] -\operatorname{softmax}\left(Y_{i+1}\right)* \log \left[\operatorname{softmax}\left(Y_{i+1}\right)\right]  & \end{aligned}$                              (7)

The accuracy level achieved by the proposed model, which uses a combination of deep learning models and a rank-based approach, is impressive, with a maximum accuracy level of 64.5% achieved on multiple datasets. This level of accuracy is higher than that achieved by individual models such as AlexNet, GoogLeNet with LSTM, GoogLeNet with Generative Adversarial Network, and GoogLeNet with attention-based encoder model, which achieve accuracies ranging from 52% to 60% when evaluated on similar datasets. The proposed model has the potential to outperform other existing models for video summarization tasks. However, it is important to note that the choice of dataset and evaluation metrics can significantly affect the performance of the model. Therefore, it is important to evaluate the proposed model on a variety of datasets and metrics to gain a comprehensive understanding of its effectiveness and limitations.

5.png

Figure 5. Coarse-grained and fine-grained deep learning models for video summarization [8]

The use of the BiLSTM model with attention-based encoder and decoder models proposed in Ji et al. [10] has shown significant improvements in accuracy for video summarization tasks, achieving an impressive accuracy of 68%. This improvement can be attributed to the high-complexity feature extraction method used in the model. Other models, such as RNN with VGGNet, have a lower accuracy of 52% when evaluated on the same dataset. Dynamic sequence deep neural network with GoogLeNet has a higher accuracy of 65% but still falls short of the accuracy achieved by the proposed BiLSTM model with attention-based encoder and decoder models. The performance of these models can vary depending on the dataset and evaluation metrics used. Therefore, it is important to evaluate these models on a variety of datasets and metrics to gain a comprehensive understanding of their effectiveness and limitations for video summarization tasks.

6.png

Figure 6. Encoder, attention, and decoder blocks for improved video feature extraction [10]

Figure 6 shows the interconnections between the many building elements that contribute to the system's high accuracy and reveals the role played by the auto-encoder module in performing high-level data augmentation. Patch-level video summarization using block sparse representation, as proposed in Mei et al. [11], can further enhance this accuracy, which is already high compared to previous methods. In this study, researchers extracted patches from input video frames and evaluated the feature values of each patch independently. With these feature values in hand, an Orthogonal Matching Pursuit (OMP) is used to find an exact fit between frames. Keyframes are taken from the video and matched based on the criteria. This methodology allows for real-time deployments with accuracies between 65% and 80%. In Yuan et al. [12], we design a new architecture that employs lengthy short-term memory models within a cycle-consistent Adversarial network. In this model, we combine two GANs in a way that allows them to learn from one another to improve their accuracy. This function for reducing errors may be derived from equation 8 using the frame difference metric (Dfi) as the final error reducer. A4 (8.27×11.69 inches) is the recommended page size for a printed book. While using the template, keep the existing page settings.

$E_r=\frac{1}{2} *\left[\sum_{i=1}^{\frac{N}{2}} p_i * D_{f_i}+\sum_{i=1+\frac{N}{2}}^N p_i * D_{f_i}\right]$                         (8)

If we say that the video is summarised across N frames, then we mean that the video is compressed over N frames. The primary goal is to minimize this mistake, which would boost video summarising accuracy from 52% to 65%, surpassing the results of approaches like Sum-based GAN (52% accuracy) and deep reinforcement learning (57% accuracy). By including the TED model proposed in Huang and Wang [13], in which a capsule net and a self-attention network are fused to pinpoint the exact moments in time when frames change, we can increase the precision of our results. Figure 7 displays capsule networks and their link with sequence self-attention networks, which are used to assess these time points via inter-frames motion curve.

7.png

Figure 7. Capsule net with sequential attention network for video summarization [13]

Architecturally, the system is designed to collect an attention curve and assign it to a key-frame block, where the video's content and motion are examined to provide a final, all-encompassing summary. The model's 91% accuracy may be attributed to the attention model, which is defined as follows in Eq. (9):

$\begin{aligned} & A_m =\left[\frac{(\partial-t)-\sin (\partial-t)}{\partial-\sin (\partial)}\right] * h_0+\left[\frac{t-\sin (t)}{\partial-\sin (\partial)}\right] * h_3+M \\ & *\left[\frac{\partial * \cos (t)-t * \cos (t)-\sin (\partial)+\sin (t)}{\partial * \cos (\partial)-\sin (\partial)}\right. \\ & \left.-\left[\frac{(\partial-t)-\sin (\partial-t)}{\partial-\sin (\partial)}\right] * h_0\right] * h_1+M \\ & *\left[\frac{\sin (\partial-t)-t * \cos (\partial)-\sin (\partial)}{\partial * \cos (\partial)-\sin (\partial)}-\left[\frac{t-\sin (t)}{\partial-\sin (\partial)}\right] * h_3\right] * h_2\end{aligned}$                                    (9)

where, h represents the H-Bézier curve's coefficients, t represents the time interval, " represents the difference between frames, and M represents the dynamic change variable specified by the following Eq. (10):

$M=\frac{(\cos \partial+1) *(\sin \partial+1)}{(\partial * \cos (\partial))+(\partial-\sin (\partial))}$                                (10)

The proposed model described earlier is highly efficient for a wide variety of datasets, outperforming other models such as shot clustering (44% accuracy), scene transition graph (54% accuracy), hierarchical clustering (70% accuracy), fast CNN (88% accuracy), and spatiotemporal CNN (81% accuracy) according to comparisons presented in the study of Huang and Wang [13]. Another approach using the Markov Decision Process (MDP) based on the optimization of the reward function described in Eq. (11) has also been proposed for video summarization in Lei et al. [14]. This model achieves a comparable high accuracy of 84.7% across multiple datasets.

$E_{\text {reward }}=\left.n^1\right|_{i=1} ^N * \sum_{\emptyset} \gamma^k * I_{j-i, k-i} * R_t\left(I_{j, k}\right)$                  (11)

where, N represents the total number of frames, p starting error probability, I represents the frame, and R represents the reward function for the frame with dimensions j by k. Large computational delays are incurred by these models; however, by processing these films on the cloud, using the elastic video summarising technique, as described in Wang et al. [15], we may eliminate these delays and achieve a more effective summary. This technique, which is very similar to CNN models, runs the full video summarising process on high-performance cloud machines, resulting in a 25% increase in total system speed and approximately 85% accuracy of summary.

The context of the video can also be taken into account while summarising it; for example, if the video is based on a sporting event, aspects such as the playing field, the players and movement, sports-specific activities, etc. can be identified and included in the summary. Time series analysis of these actions can be utilized to enhance video summary results. Using a perceptual retrieval model based on feature optimization, Thomas et al. [16] provides a model for such context-based video retrieval, which achieves an accuracy level of 71% on a wide set of movies. For even higher precision, it is recommended to apply powerful feature selection techniques, such as principle component analysis (PCA) and Linear Discriminant Analysis (LDA), as discussed in Raksha et al. [17]. With an accuracy of 93.7%, sufficient for real-time video summarization systems, these techniques attempt to reduce input size by extracting maximal variance features, which are subsequently used for classification using CNN. Further evaluation of system performance and deploy ability requires testing this accuracy on numerous datasets. In Jiang et al. [18], we find a discussion of a different deep learning model that employs multi-task learning with CNN to get an accuracy of 82%. In Xiao et al. [19], for example, the authors combine local attention models with a global attention model to boost classification precision. Figure 8 depicts the model's architecture, which involves splitting an input video into several frames and then clustering each of those frames into related segments.

8.png

Figure 8. Multiple attention models for video summarization [19]

Table 1. Accuracy and area of application-based comparison for different models