Image Content Analysis for Social Media Public Opinion Monitoring and Response Strategies

With the widespread application of social media, the formation and dissemination speed of online public opinion have greatly accelerated, and the influence of public opinion events has also increased significantly [1, 2]. In this context, how to effectively monitor and manage public opinion information on social media platforms has become a focal point of concern for society [3-6]. Social media not only carries a large amount of text information but also includes a large number of images, videos, and other multimedia content, which play an increasingly important role in the dissemination of public opinion events [7, 8]. Therefore, how to use image content analysis techniques to deeply mine public opinion information on social media has become an urgent problem in the field of public opinion monitoring.

The significance of research on public opinion monitoring and response lies in its ability to help governments, enterprises, and other relevant departments understand the emotional changes and public opinion trends of society in a timely manner, and also provide a scientific basis for crisis management. Traditional public opinion monitoring methods mostly focus on the analysis of text information, neglecting the potential value of image content [9-13]. However, in the current social media environment, images and videos often convey emotions and information that are more intuitive and richer than text. Especially in the dissemination of emergencies or sensitive topics, image content can quickly attract widespread attention [14-18]. Therefore, research on image content-based public opinion monitoring methods can not only improve the accuracy of public opinion monitoring but also provide more comprehensive decision-making support for public opinion management and response.

Although some progress has been made in the field of public opinion monitoring, most studies are still limited to the processing of text information, lacking in-depth analysis of image content. In addition, existing image content analysis methods often focus on basic feature extraction of images, without fully mining the potential emotional tendencies and social influence within the images [19-24]. Moreover, traditional public opinion monitoring models often have biases in their use of semantics, leading to insufficient accuracy and effectiveness of monitoring results. Therefore, existing research methods are difficult to meet the practical application needs in the face of the complex and dynamic social media environment.

This paper aims to explore image content-based social media public opinion monitoring and response strategies. The research mainly includes three parts: First, analyzing the issues related to the improper use of semantics in traditional public opinion monitoring methods and proposing improvement strategies; second, constructing an image content analysis model for social media public opinion monitoring to improve the precision of image information interpretation and emotional judgment capability; third, based on the results of image content analysis, proposing corresponding public opinion response strategies for social media, to help relevant institutions respond to public opinion crises in a timely and effective manner. This study will not only help improve the accuracy of public opinion monitoring technology but also provide governments and enterprises with more scientific public opinion response solutions, with important theoretical value and practical significance.

3.1 Model architecture overview

The model proposed in this paper is designed with a decoupled structure, which is its key advantage. By separating the visual decoder and the semantic decoder, the model is able to independently extract pure visual features and semantic features. This decoupling design effectively reduces the dependency on the vocabulary and avoids the negative impacts caused by limited corpus and noise interference. The visual decoder focuses on extracting visual information from the image, while the semantic decoder captures global semantic information through bidirectional encoding, ensuring the accuracy and comprehensiveness of the information. Additionally, self-supervised pretraining tasks, such as word spelling correction, are introduced to further enhance the semantic modeling capabilities of the semantic module. By utilizing a large corpus of freely available vocabulary, the model continuously optimizes its semantic understanding during training. This innovative training method not only reduces reliance on high-quality annotated data but also improves the model's performance in the complex social media environment. Figure 2 illustrates the architecture of the proposed visual-semantic decoupled image content analysis model.

Specifically, the model is based on the Aster framework with improvements to meet the needs of social media image content analysis. Targeted optimizations have been made to the architecture. First, the model uses a Spatial Transformer Network (STN) module based on Thin Plate Spline (TPS) transformations to geometrically correct the input images, addressing the issue of arbitrary-shaped text found in social media images and ensuring effective recognition of text lines in various formats. Next, the visual feature extractor, which includes a 45-layer ResNet and two layers of bidirectional LSTM, extracts features from the corrected image, producing an original visual feature sequence g⁽ⁿ⁾ with a length of S and dimensionality of F, providing efficient visual information for subsequent semantic modeling.

In the decoding phase, the decoupling of the visual and semantic information decoders enables the model to more efficiently handle the complex emotions and potential public opinion information embedded in social media images. In the visual decoding part, the model first processes the raw visual feature sequence g⁽ⁿ⁾ through a visual encoder, then passes these features to the visual decoder for stepwise decoding. To ensure that the visual features of each character are accurately extracted and aligned with the corresponding semantic information, the model introduces an innovative mechanism: when calculating the alignment of character features, the visual decoder relies not only on the visual features of the current step but also incorporates the hidden state from the semantic module. The core of this design is to leverage the semantic module’s assistance to help the visual decoder locate and decode accurate characters in the highly variable social media images, while ensuring that the visual features t⁽ⁿ⁾ for each character are derived from the weighted average of the original visual feature sequence, maintaining the purity of the visual information and avoiding interference from image noise or irregular expressions. Figure 3 illustrates the decoding process of the visual feature decoder.

2.png

Figure 2. Architecture of the visual-semantic decoupled image content analysis model

3.png

Figure 3. Decoding process of the visual feature decoder

In the semantic modeling part, model V decouples the processing of semantic features from the visual features. To capture the emotions, implicit social intentions, and potential cultural background behind the image text, the model first maps the raw text sequence S into an initial semantic feature sequence g^(t) through a character embedding layer, and then uses a semantic encoder to extract contextual features from g^(t), thus modeling global semantic information. On this basis, the semantic decoder, as a serial character-level language model, further processes each step's prediction. In the decoding process, the semantic decoder not only utilizes global semantic information but also combines the embedding of the previously predicted character to form a more accurate text output. Through this design, the model avoids the strong coupling of semantic and visual information found in traditional models, independently modeling semantic features. This overcomes the limitations of the training corpus vocabulary and enables the model to flexibly respond to the rich, diverse, and non-standardized images and text content on social media, accurately identifying complex public opinion information.

In social media public opinion monitoring, images usually contain multi-layered, diversified content, involving emotions, implicit social viewpoints, or cultural backgrounds. These types of information may not be fully expressed through visual or semantic features alone. Therefore, during the recognition phase, the model concatenates the t⁽ⁿ⁾ and t^(t) features extracted by the visual and semantic decoders, respectively, to form a richer feature representation, which is then passed to the character classification layer for processing. The concatenated feature combination captures not only the visual information in the image but also effectively integrates the contextual knowledge at the semantic level, ensuring accurate text recognition when facing potential emotional cues or social backgrounds in social media images. In the character classification phase, the concatenated visual and semantic features are used as input to a character classification layer. At this point, the decoding process is serial, where each step is based on the previous character prediction and the current visual-semantic features for decision-making. In each decoding step s, the model first produces a probability distribution O_s, where Z represents the number of categories, indicating the probability of the current character belonging to each category. Then, the model selects the category index corresponding to the highest probability to obtain the character prediction result b_s.

3.2 Visual feature decoder

Social media images often contain a mixture of visual elements, such as text, emojis, image backgrounds, and specific cultural symbols. The visual feature decoder in this model must be capable of handling these complex and diverse inputs. In the decoding process, the visual feature decoder first concatenates the already obtained semantic features t^(t)_s and the embedding of the current position r^(o)_s, generating the query vector for the current step. Then, by combining these query vectors with the original visual feature sequence g⁽ⁿ⁾, the attention mechanism calculates the attention weights β_s, which determine the visual regions that need to be focused on for the current character.

$z_{s, u}=Q^{S \tanh }\left(I\left[t_s^{(t)} ; r_s^{(o)}\right]+N g_u^{(n)}+y\right)$                   (3)

$\beta_{s, u}=\frac{\exp \left(z_{s, u}\right)}{\sum_{u u^{\prime}=1}^S \exp \left(z_{s, u^{\prime}}\right)}$                (4)

The above process is essentially a character alignment operation, where the original visual feature sequence is weighted using a soft mask to highlight the visual features related to the current decoded character, ensuring accurate extraction of the complex visual information in social media images. With this attention mechanism, the model is able to focus on the most relevant visual content at each decoding step, whether handling clear text or more complex or blurry scenes in the image. Clearly, β_s can also be seen as a soft mask of the specific location of the character to be decoded in g⁽ⁿ⁾. The expressions for the original and new visual features of the character are as follows:

$h_s=\sum_{u=1}^S \beta_{s, u} t_u$                          (5)

$t_s^{(t)}=\operatorname{GRUCell}\left(t_{s-1}^{(t)}, h_s\right)$                        (6)

3.3 Semantic module

In traditional visual-semantic fusion models, semantic information is often directly extracted from the original visual features. However, the semantic module in this model starts with the text itself for encoding, avoiding interference from visual noise and improving the accuracy of global semantic modeling. This approach is especially beneficial in social media content, where the emotional tone, context, and implied meaning of the text are often very complex. By leveraging a language model-based structure, the semantic module is better equipped to handle these challenges. Figure 4 illustrates the decoding process of the semantic feature decoder. In the proposed model, the semantic module processes the text features independently using a character-level self-encoding language model, providing precise semantic understanding of the text content in social media images. The module's Connectionist Temporal Classification (CTC) decoder first makes an initial prediction of the text in the social media image, obtaining a predicted text sequence. This sequence is then mapped to the raw semantic feature sequence g^(t) through a character embedding layer. The character embedding not only maps each character directly but also helps the subsequent semantic encoder better understand the text's language structure and semantic content. The semantic encoder processes g^(t) by inputting it into a two-layer bidirectional Gated Recurrent Unit (GRU), capturing the contextual information in the text and establishing a comprehensive understanding of the input text at the global semantic level. Further, the semantic module integrates the sequence features extracted from the bidirectional GRU through a fully connected layer to produce a feature vector t_h, which encapsulates the global semantic information of the input string. The obtained feature vector t_h not only provides a holistic understanding of the text but also helps the model capture underlying emotional information, public opinion trends, and social intentions. Finally, the semantic decoder further maps the global semantic information t_h to the initialized semantic features t^(t)₀ and the semantic encouragement features e at each step. These features guide the prediction of each character during the decoding process, improving the accuracy of text recognition in the public opinion monitoring task, especially when analyzing image content with specific emotions, viewpoints, or cultural backgrounds. Assuming the embedding of the character predicted at step s−1, is denoted as b_s-1, the semantic features t^(t)_s for step s are computed as follows:

$t_s^{(t)}=\operatorname{GRUCell}\left(t_{s-1}^{(t)},\left[d\left(b_{s-1}\right) ; e\right]\right)$                        (7)

4.png

Figure 4. Decoding process of the semantic feature decoder

3.4 Loss function

The optimization objective function during the training phase of the proposed model consists of four components, each corresponding to the CTC decoder, visual decoder, semantic decoder, and the final classification layer. Each decoder independently outputs string predictions through its own character classification layer, helping the model to perform supervised learning at multiple levels. Let the CTC loss function be represented as LOSS_CTC, and the cross-entropy loss functions for the visual decoder, semantic decoder, and the classification layer be represented as LOSS_SJ, LOSS_YY, and LOSS_FL, respectively. The hyperparameter for these losses is denoted by η, and the overall loss function can be expressed as:

$L O S S=\eta_{C T C} L O S S_{C T C}+\eta_{S J} L O S S_{S J}+\eta_{Y Y} L O S S_{Y Y}+\eta_{F L} L O S S_{F L}$                (8)

3.5 Comparison with other semantic-enhanced models

The visual-semantic feature decoupling model proposed in this paper offers unique advantages in the image content analysis of social media for public opinion monitoring, especially in character localization and text semantic modeling. This model is particularly effective in addressing common challenges in social media environments, such as blurred images, text overlap, and dense characters. Compared to other semantic-enhanced models, the model presented in this paper explicitly decouples visual and semantic features, processing and optimizing these two modalities independently through separate visual and semantic decoders. This decoupling design is especially important in social media public opinion monitoring tasks, as the text in social media images often contains multi-layered emotional expressions and complex social intentions. Traditional visual-semantic coupled models, such as DAN and SRN, typically rely on visual features for character alignment or localization. While these models may perform well in simple scenarios, their single visual feature-based localization approach is insufficient for handling complex situations in social media images, such as dense text, overlapping characters, or blurry scenes. In particular, during public opinion monitoring, images often carry complex information related to emotions and social intentions. The proposed model improves character alignment using more accurate semantic features, ensuring precise location prediction and avoiding misjudgments and information loss. This enhancement boosts the model's ability to understand complex image content effectively.

In social media public opinion monitoring applications, especially for tasks like sentiment analysis and public opinion trend identification, semantic information modeling is crucial. The SRN model, which relies solely on visual features for character alignment, can handle simple text recognition tasks but struggles in social media images, especially when text spacing is small or image quality is poor. For example, social media images often contain user-generated content with strong emotional tones, influenced by individual emotions or social events. The SRN’s reliance on single visual features leads to poor performance in these complex contexts. By introducing the semantic decoder and adopting a decoupled bimodal framework, the proposed model integrates semantic information into the character alignment stage, enabling the model to combine visual data and semantic context. This integration not only improves prediction accuracy but also enhances the model’s ability to capture the underlying emotions and social intentions in the text. Thus, in social media public opinion monitoring, the model can more accurately analyze the emotional tendencies, social attitudes, and potential public opinion shifts in image text, providing strong support for subsequent public opinion response strategies.

From the data in Table 1, it can be observed that the accuracy of word auto-encoding using the FastText model significantly varies with different acquisition methods. On the training set, the combination of initialization (init) and step-by-step training (step) (init & step) performs the best, with an accuracy of 88.9%, far surpassing the other two methods (init and step, which are 52.3% and 42.5%, respectively). For the test set, Init & step also shows a significantly higher accuracy compared to the other two methods, achieving 88.5%, while init and step only reached 7.3% and 6.2%, respectively. This indicates that the combination of initialization and step-by-step training can significantly improve the model's generalization ability on the test set, and that the handling of semantic features for social media public opinion monitoring is particularly crucial in image content analysis tasks.

Table 1. The impact of utilization and acquisition methods of global semantic features in social media public opinion on word auto-encoding accuracy

Table 2. The impact of human involvement frequency in word error correction on underfitting in word error correction learning

From the data in Table 2, it can be seen that the frequency of human involvement in word error correction significantly affects the accuracy of word error correction. When the frequency of human correction is 100%, the accuracy is 58.8%. However, when the human involvement frequency is reduced to 5%, the accuracy significantly increases to 75.4%. This result suggests that, although human correction improves accuracy to some extent, as the frequency of human intervention decreases, the model's learning ability and automatic error correction ability are better utilized, leading to an improvement in accuracy. This could be because human intervention, in some cases, limits the model's space for autonomous learning, which affects its overall understanding and adaptability to public opinion content.

From the comparison of public opinion recognition accuracy across different models in Table 3, it can be seen that the proposed model outperforms other traditional models on multiple datasets. Specifically, the proposed model shows high accuracy across several datasets, including Flickr30k (93.8%), Twitter Image Dataset (92.9%), Social Media Image Dataset (94.3%), ImageNet (84.2%), Memes Dataset (85.5%), and Open Images Dataset (87.2%). Notably, on the Social Media Image Dataset and Flickr30k datasets, the accuracy is 94.3% and 93.8%, respectively, both leading among all models. This demonstrates that the proposed model exhibits excellent semantic understanding and sentiment judgment capabilities across various social media image datasets, enabling effective recognition and classification of public opinion information in social media images.

Figure 5 presents recognition samples of the proposed model, demonstrating its outstanding performance in image content analysis, especially in the context of social media public opinion monitoring. The model efficiently identifies and classifies complex image information. Testing on different datasets shows that the model excels at recognizing difficult characters in key social media public opinion words, whether they appear at the beginning or in the middle of the word. Additionally, while identifying social media images' public opinion information, the model also exhibits strong strategy generation capabilities, effectively leveraging global semantic information, not just unidirectional semantics. This makes it easier for decision-makers to respond more scientifically and promptly during public opinion crises.

According to the data in Table 4, the loss functions in the visual and semantic decoders significantly impact model training. The proposed model achieves high accuracy when using the LOSS_SJ loss function, with a training set accuracy of 92.3% and a test set accuracy of 92.6%. Using the LOSS_YY loss function results in training and test set accuracies of 92.5% and 92.5%, respectively, slightly outperforming the LOSS_SJ function. Combining both loss functions yields the best performance, with accuracies of 92.1% for the training set and 92.8% for the test set. This demonstrates that combining visual and semantic loss functions effectively enhances model training and achieves high accuracy on the test set.

5.png

Figure 5. Recognition samples of the proposed model

Table 4. The role of loss functions in the visual and semantic decoders on model training

These experimental results further confirm the critical role of loss functions in image content analysis models for social media public opinion monitoring. Specifically, the combination of LOSS_SJ and LOSS_YY optimizes the model's performance in visual and semantic feature decoding, enabling it to better interpret image content and assess sentiment, thus improving public opinion monitoring accuracy. The proposed model, by optimizing the relationship between image and semantic features, avoids the common issues of improper semantic usage found in traditional public opinion monitoring systems.

Table 5. The impact of different query requests on social media public opinion recognition accuracy

The data in Table 5 shows the impact of different query requests on social media public opinion recognition accuracy. Specifically, it compares the effects of "Previous Visual Hidden Layer State" and "Current Semantic Features" on model accuracy in the training and test sets. In the training set, using "Current Semantic Features" improves accuracy to 92.6%, compared to 91.2% when using "Previous Visual Hidden Layer State." However, in the test set, the "Previous Visual Hidden Layer State" yields better accuracy (82.5%) than "Current Semantic Features" (81.9%), indicating that the model relies more on visual information when processing the test set, while the influence of semantic features is slightly weaker.

These results suggest that visual and semantic features do not play equal roles in social media public opinion monitoring, with each affecting the model's training and testing accuracy differently. Semantic features contribute more significantly in the training set, likely because they help the model better understand the emotions and sentiments behind the image content, thus improving its ability to recognize complex social media public opinion. In contrast, in the test set, the performance of visual hidden layer states is more pronounced, likely due to the complexity of the image content in the test set, where the model relies more on direct visual features for judgment. In practical applications, the combination of visual and semantic information should be adjusted based on the specific task and data characteristics.