A Framework for Tourist Behavior Recognition and Interest Region Generation in Digital Cultural Tourism Platforms

2.1 Overall framework architecture

The proposed TBA-IRNet adopts an encoder–decoder architecture, in which surveillance video streams from cultural tourism scenarios are taken as input. The overall design is centered on shared feature extraction, parallel multi-task inference, bidirectional attention interaction, and end-to-end optimization. The encoder is constructed based on a three-dimensional convolutional backbone network, through which multi-scale spatiotemporal features are extracted from consecutive video frames. These features provide a unified and highly expressive representation for subsequent dual-task inference, thereby mitigating feature redundancy and information fragmentation. The spatiotemporal feature maps generated by the encoder are simultaneously fed into two parallel task branches, which are responsible for tourist behavior recognition and interest region generation, respectively. These branches are not independently operated; instead, bidirectional information exchange is enabled through a collaborative attention mechanism. In this manner, tourist behavior recognition is enhanced by incorporating region-level semantic information to improve classification accuracy, while interest region generation is guided to focus on behaviorally active tourist groups. As a result, mutual reinforcement between the two tasks is achieved. The entire framework is designed to be fully differentiable, allowing joint training and optimization of the shared backbone and both task branches through a unified multi-task loss function. End-to-end inference is thus realized, spanning from raw video input to the outputs of behavior categories and interest regions. This design effectively improves overall model performance and inference efficiency, while ensuring robustness in complex cultural tourism environments characterized by high crowd density and diverse behavioral patterns.

2.2 Detailed description of core modules

2.2.1 Multi-task collaborative interactive attention mechanism

To eliminate the task separation between tourist behavior recognition and interest region generation and to achieve synergistic enhancement between the two tasks, a multi-task collaborative interactive attention module is designed. The core innovation of this module lies in the construction of a bidirectional information flow mechanism between the two tasks. In contrast to conventional unidirectional attention guidance schemes, mutual feature enhancement and dynamic optimization are enabled, while full differentiability is preserved to support end-to-end joint training. The overall architecture of the module is illustrated in Figure 1. Based on the spatiotemporal feature maps extracted by the shared backbone network, dedicated interaction pathways are established between the tourist behavior recognition branch and the interest region generation branch. No additional redundant feature extraction modules are introduced, thereby improving feature utilization efficiency and effectively addressing the fundamental limitation of task-wise information fragmentation observed in existing methods.

Figure 1. Network architecture of the multi-task collaborative bidirectional interactive attention mechanism

The bidirectional information flow mechanism constitutes the core technical innovation of this module, and its implementation is described below. When behavioral semantic information is transmitted from the tourist behavior recognition branch to the interest region generation branch, the initial feature representations of the behavior recognition branch are first subjected to a linear projection. Through this transformation, high-dimensional features are mapped into attention weights that are dimensionally aligned with the feature maps of the interest region generation branch. The projection process can be formulated as:

$F_{a c t ~ p r o j}=W_{a c t} \cdot F_{a c t}+b_{a c t}$     (1)

where, F_act denotes the initial feature representation of the tourist behavior recognition branch, W_act and b_act represent the learnable projection weight matrix and bias vector, respectively, and F_{act_proj} denotes the resulting attention weight map after projection. Subsequently, a Sigmoid activation function is applied to normalize the attention weights, yielding attention coefficients within the range [0, 1]. These coefficients are then multiplied element-wise with the feature maps of the interest region generation branch. In this manner, behavioral semantics are effectively injected into the region generation process, enabling the generated interest regions to focus on behaviorally active tourist groups and thereby improving the semantic accuracy of interest region generation. Conversely, when contextual information is transmitted from the interest region generation branch to the tourist behavior recognition branch, candidate region features are first extracted from the interest region generation branch. These features are similarly transformed via a linear projection to obtain context-constrained vectors. The resulting representations are then integrated into the node feature aggregation process of the graph convolutional network within the behavior recognition branch, thereby enhancing the contextual coherence of node features. The projection and fusion process can be expressed as:

$F_{\text {node_att }}=F_{\text {node }}+\alpha \cdot\left(W_{i r} \cdot F_{i r_{-} c a n d}+b_{i r}\right)$   (2)

where, F_{ir_cand} denotes the candidate interest region features, W_ir and b_ir represent the corresponding projection parameters, α is a weighting coefficient controlling the contribution of contextual fusion, F_node denotes the initial node features in the graph convolutional network, and F_{node_att} represents the node features after integration of region-level contextual information. This process improves the accuracy of tourist behavior classification.

The fully differentiable design of the module constitutes a critical innovation for enabling end-to-end joint optimization. To address the issue of training discontinuities caused by non-differentiable operations in conventional attention mechanisms, all components of the proposed module—including feature projection, attention weight computation, and feature fusion—are implemented using differentiable operators. Linear projections are realized through learnable parameter matrices. Attention weight normalization is performed using the Sigmoid differentiable activation function. Feature fusion is achieved through element-wise multiplication and linear combination, both of which are differentiable operations. As a result, gradient backpropagation is preserved throughout the entire interaction process. This design enables the shared backbone network, the tourist behavior recognition branch, the interest region generation branch, and the interactive attention module to be jointly trained under a unified loss function. Consequently, information loss during transmission is mitigated, while training stability and computational efficiency are improved. The synergistic benefits of the dual-task framework are thereby fully realized.

2.2.2 Interest region generation method with behavior-aware weighting

To address the fundamental limitation of conventional interest region generation methods, which rely solely on crowd density while neglecting behavioral semantics and temporal information [17, 18], an interest region generation method with behavior-aware weighting is proposed. The core innovation of this approach lies in the integration of tourist behavior categories, dwell time, and individual-scale features into the modeling of interest intensity, thereby enabling semantically meaningful and precise generation of interest regions. Meanwhile, full differentiability is preserved throughout the generation process, ensuring compatibility with end-to-end joint training. Specifically, the proposed method constructs an initial interest intensity map that accurately reflects tourists’ interest preferences through three key components: behavior weight assignment, temporal decay factor design, and adaptive Gaussian kernel construction. These components provide accurate semantic guidance for subsequent region proposal generation. A schematic illustration of the interest region intensity map generation process based on behavior-aware weighting is presented in Figure 2.

The behavior weight assignment strategy is designed to capture the semantic differences among tourist behaviors in cultural tourism scenarios. An adaptive weighting mechanism is introduced, in which weight levels are determined according to the indicative strength of each behavior with respect to interest regions. Specifically, photographing behavior is assigned the highest weight, followed by dwelling behavior, while walking behavior is assigned the lowest weight. The weight values are dynamically calibrated through five-fold cross-validation to ensure adaptability across different cultural tourism environments, thereby enhancing both rationality and generalization capability. A temporal decay factor is introduced to distinguish between short-term and long-term dwell behaviors, thereby preventing transient passersby from being incorrectly identified as core contributors to interest regions. The decay function is constructed based on the number of consecutive dwelling frames for each tourist, and is defined as:

$d_i^{\prime}=1+\log \left(1+d_i\right)$     (3)

where, d_i denotes the number of consecutive dwelling frames of the i-th tourist prior to the current frame, and d_i′ represents the normalized temporal decay factor. The logarithmic function effectively suppresses the imbalance caused by rapid increases in frame counts, allowing the contribution of long-term dwellers to be reasonably amplified while mitigating the influence of short-term visitors. Furthermore, an adaptive Gaussian kernel is designed to address the mismatch between fixed kernel sizes and varying tourist scales in conventional methods. The standard deviation of the Gaussian kernel is dynamically adjusted according to the size of each tourist’s bounding box, and is computed as:

$\sigma=0.3 \times \max \left(w_i, h_i\right)$  (4)

where, w_i and h_i denote the width and height of the i-th tourist bounding box, respectively. This design ensures that the spatial coverage of the Gaussian kernel is well aligned with the actual scale of each tourist, thereby avoiding spatial bias introduced by fixed kernel sizes and improving the spatial accuracy of the generated interest intensity map.

Based on the aforementioned design, the interest intensity map is generated by aggregating the weighted Gaussian kernels of all tourists. The weighted Gaussian kernel corresponding to an individual tourist is defined as:

$H_i(x, y)=w_c \cdot d_i^{\prime} \cdot \exp \left(-\frac{\left(x-x_i\right)^2+\left(y-y_i\right)^2}{2 \sigma^2}\right)$  (5)

where, w_c denotes the weight associated with the tourist behavior category, (x_i,y_i) represents the center coordinates of the tourist bounding box, and (x,y) denotes the pixel coordinates in the image. The initial interest intensity map for the entire image is obtained by summing the weighted Gaussian kernels of all tourists at the pixel level. The resulting pixel values are positively correlated with the degree of tourist interest at corresponding spatial locations, thereby enabling precise characterization of the interest distribution in the scene based on behavioral semantics. The entire generation process is implemented using differentiable operations, ensuring seamless integration with other components of the framework during joint training. The generated interest intensity map not only encodes crowd density information but also incorporates behavioral semantics and temporal characteristics. Consequently, the semantic accuracy and practical relevance of subsequent interest region proposals are significantly improved.

2.2.3 Bidirectional guidance mechanism of spatiotemporal graphs and region-level attention

To accurately model the spatiotemporal interaction characteristics of tourists in cultural tourism scenarios, while achieving semantic alignment between tourist behavior recognition and interest region generation, a bidirectional guidance mechanism integrating spatiotemporal graphs and region-level attention is proposed. The core innovation of this mechanism lies in the construction of a fine-grained spatiotemporal graph model and a bidirectional attention interaction pathway. This design effectively addresses the limitations of conventional graph-based methods, in which region-level semantic information is insufficiently incorporated, and attention mechanisms are typically constrained to unidirectional guidance. As a result, mutual enhancement between the two tasks is achieved, leading to improved overall performance. Specifically, the refined spatiotemporal graphs constructed and a graph convolutional framework enhanced by multi-head attention designed provide a robust feature foundation for bidirectional guidance, thereby ensuring both the precision and effectiveness of the guidance process.

Figure 3. Schematic diagram of spatiotemporal graph construction and bidirectional guidance via region-level attention

As illustrated in Figure 3, the proposed mechanism for spatiotemporal graph construction and bidirectional guidance via region-level attention is presented. The fine-grained construction of the spatiotemporal graphs is centered on the innovative design of both node and edge features, thereby overcoming the limitations of insufficient semantic expressiveness in conventional graph structures. Node features are constructed using a multi-dimensional feature concatenation strategy, in which tourist appearance features, normalized pose keypoint coordinates, and bounding box positional features are integrated to comprehensively capture individual attributes and spatial information. Among these, pose keypoint features are incorporated to facilitate the representation of subtle behavioral variations, thereby enhancing the semantic expressiveness of node representations for tourist behavior recognition. Edge features are designed to jointly capture spatial interactions and temporal dynamics. Intra-frame edges are established based on a Euclidean distance threshold between tourists, which is determined through statistical analysis of the dataset and set to 50 pixels. Specifically, an intra-frame edge is created when the Euclidean distance between two tourists is less than this threshold, enabling the modeling of spatial interaction relationships. Inter-frame edges are constructed by associating nodes of the same tourist across consecutive frames using tracking identifiers, thereby capturing temporal dynamics at the individual level. In addition, relative distance and directional information are incorporated into edge features to further enrich interaction semantics. On this basis, a multi-head attention mechanism is introduced into the spatial graph convolutional operation. Four attention heads are employed, each of which adaptively learns neighborhood weights under different interaction patterns. The attention weight is computed as:

$\alpha_{i j}=\frac{\exp \left(\operatorname{LeakyReLU}\left(W_q h_i^T W_k h_j\right)\right)}{\sum_{k \in N(i)} \exp \left(\operatorname{LeakyReLU}\left(W_q h_i^T W_k h_k\right)\right)}$       (6)

where, $\alpha_{i j}$ denotes the attention weight assigned by node $i$ to its neighboring node $j ; h_i$ and $h_j $ represent the features of nodes $i$ and $j$, respectively; $W_q$ and $W_k$ are learnable parameter matrices, and $N(i)$ denotes the set of neighboring nodes of node $i$. This design addresses the limitation of fixed neighborhood weights in conventional spatiotemporal graph convolution methods and enhances the model's capability to capture complex group behaviors, such as crowd aggregation and collective attention patterns.

The bidirectional guidance mechanism constitutes the core innovation of this module, through which mutual constraint and optimization between tourist behavior recognition and interest region generation are achieved. When region-level attention is used to guide tourist behavior recognition, the attention map generated by the interest region generation branch is first extracted and then transformed into a neighborhood weight adjustment factor for graph convolution through a linear projection. The projection process is defined as $\beta=W_\beta A_{i r}+b_\beta$, where A_ir denotes the interest region attention map, W_β and b_β represent projection parameters, and β denotes the adjustment factor. This factor is multiplied with the neighborhood weights in the graph convolutional operation, thereby enabling node feature aggregation to focus more effectively on tourist interactions within interest regions, which leads to improved accuracy in tourist behavior recognition. Conversely, when behavioral features are used to guide interest region generation, node features from the tourist behavior recognition branch are projected into the query vector Q, while intermediate features from the region proposal network are treated as key K and value V. Cross-attention is then employed to incorporate behavioral semantics into region generation. The cross-attention operation is defined as:

$\operatorname{Att}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$     (7)

where, d_k denotes the dimensionality of the key vectors. Through this operation, behavioral semantics are effectively embedded into region-level features, thereby improving the confidence prediction of candidate regions and filtering out redundant regions associated with behaviorally inactive areas. As a result, the generated interest regions are ensured to be highly consistent with tourist behavior semantics. Through this bidirectional guidance mechanism, deep interaction between spatiotemporal graph features and region-level attention features is achieved. Consequently, tourist behavior recognition and interest region generation are jointly optimized, leading to substantial improvements in model adaptability in complex cultural tourism scenarios.

2.2.4 Fully differentiable end-to-end optimization framework

To enable coordinated training and performance enhancement across all modules, and to address key limitations of existing methods—such as partial non-differentiability, suboptimal loss function design, and limited generalization capability [19, 20]—a fully differentiable end-to-end optimization framework is proposed. The core innovation lies in the design of a multi-task joint loss function tailored to cultural tourism scenarios. In addition, a novel consistency loss and an improved density map loss are introduced, together with task-specific training strategies, to ensure full gradient backpropagation throughout the pipeline and to achieve joint optimization of overall model performance. Within this framework, the shared backbone network, dual-task branches, and all innovative modules are integrated into a unified training system. Training discontinuities are thereby eliminated, allowing the synergistic advantages of all components to be fully exploited.

The design of the multi-task loss function constitutes the central component of the end-to-end optimization framework. Through careful adaptation and refinement of individual loss terms, balanced learning and semantic alignment across tasks are achieved. Considering the characteristics of cultural tourism datasets, five-fold cross-validation is employed to optimize the weighting coefficients of each loss term. The final weights are determined as λ₁=0.2, λ₂=0.1, λ₃=0.3, λ₄=0.2, λ₅=0.1, and λ₆=0.1. This configuration prioritizes the core tasks of tourist behavior recognition and interest region generation, while maintaining an appropriate balance with auxiliary tasks such as detection and pose estimation. A consistency loss is introduced to enforce semantic alignment between tourist behavior recognition and interest region generation. By computing the Kullback–Leibler (KL) divergence between the graph attention weights of the behavior recognition branch and the attention maps of the interest region generation branch, the discrepancy between the feature distributions of the two tasks is quantified. The formulation is expressed as follows:

$L_{\text {cons }}=\sum_{i=1}^N p_i \log \left(\frac{p_i}{q_i}\right)$      (8)

where, p_i denotes the attention weight distribution from the tourist behavior recognition branch, and q_i denotes the attention weight distribution from the interest region generation branch. This loss encourages consistency between the feature distributions of the two tasks, thereby promoting bidirectional collaborative enhancement. An improved density map loss is further introduced to address the limitation of conventional density-based losses, which typically ignore semantic information. The ground-truth interest density map is constructed by combining manually annotated interest regions with trajectory clustering of tourists, followed by Gaussian smoothing. The L2 distance between the predicted interest intensity map and the ground-truth density map is then computed as:

$L_{\text {density }}=\frac{1}{H \times W} \sum_{x=1}^H \sum_{y=1}^W(S(x, y)-\widehat{S}(x, y))^2$      (9)

where, $H$ and $W$ denote the height and width of the image, respectively, $S(x, y)$ denotes the ground-truth pixel value of the interest density map, and $\widehat{S}(x, y)$ denotes the predicted pixel value of the interest density map. By integrating all loss components, the overall loss function is formulated as:

$L=\lambda_1 L_{\text {det }}+\lambda_2 L_{\text {pose }}+\lambda_3 L_{\text {act }}+\lambda_4 L_{\text {region }}+\lambda_5 L_{\text {densitv }}+\lambda_6 L_{\text {cons }}$       (10)

Further optimization of the training strategy is performed to enhance model generalization and to better align with the characteristics of cultural tourism surveillance video scenarios. The backbone network is initialized using pretrained weights, and only the higher-level convolutional layers are fine-tuned. In this manner, the general feature extraction capability of lower layers is preserved, thereby mitigating overfitting and improving training efficiency. The data augmentation strategy is specifically adapted to the complexity of cultural tourism environments. Techniques, including random cropping, horizontal flipping, color jittering, and temporal frame shuffling, are employed to increase data diversity. These operations effectively alleviate the limitations imposed by the relatively small scale of the self-constructed dataset and improve the model’s robustness to variations in illumination, viewpoint, and crowd density. The entire training process is conducted under a unified optimization objective, in which all modules are designed to be fully differentiable. Consequently, gradients can be propagated seamlessly throughout the network, enabling coordinated training of the shared backbone network and dual-task branches. This design significantly enhances overall model performance and generalization capability, thereby ensuring suitability for real-world deployment in complex cultural tourism scenarios.

2.3 Inference pipeline

The inference pipeline of TBA-IRNet is characterized by its fully differentiable end-to-end design, enabling efficient and coordinated inference from surveillance video stream input to the outputs of tourist behavior categories and interest regions. No intermediate manual intervention or feature transformation is required, thereby significantly improving both inference efficiency and accuracy. The inference process begins with the input of continuous frames from cultural tourism surveillance video streams. Multi-scale spatiotemporal features are first extracted from the input frames by the shared backbone network, producing a spatiotemporal feature map with strong semantic representation capability. The feature map provides a unified feature foundation for both task branches, thereby avoiding feature redundancy and repeated extraction, and establishing the basis for efficient inference.

Based on the extracted spatiotemporal features, tourist detection and pose estimation are first performed. Structured tourist instances are then generated, including bounding boxes, pose keypoints, appearance features, and unique tracking identifiers for each individual. These structured representations provide fine-grained individual-level information for subsequent dual-task inference. The pipeline then proceeds to the parallel inference stage of the two tasks. In the tourist behavior recognition branch, a fine-grained spatiotemporal graph is constructed based on the structured tourist instances. Behavior classification is performed using a multi-head attention-enhanced spatiotemporal graph convolutional network, yielding behavior categories for each tourist. In the interest region generation branch, an initial interest intensity map is generated based on behavior-aware weighting, dwell time, and adaptive Gaussian kernels. This map is combined with the spatiotemporal feature map and processed through a differentiable region proposal network to produce candidate interest regions. Throughout this process, real-time bidirectional information exchange between the two branches is achieved via the collaborative attention mechanism, without requiring additional feature mapping steps. Finally, optimal interest regions are selected through non-maximum suppression, and the results are output together with the corresponding tourist behavior recognition results. The entire pipeline is implemented using differentiable operations, thereby eliminating inference discontinuities caused by non-differentiable components in conventional approaches. As a result, smooth feature propagation across modules is ensured, inference latency is significantly reduced, and accuracy is maintained through the bidirectional guidance mechanism. This design enables a synergistic improvement in both efficiency and precision, making the framework well-suited for real-time processing requirements in complex cultural tourism surveillance scenarios.

A detailed analysis of the experimental results indicates that the significant performance improvements achieved by the proposed TBA-IRNet framework can be primarily attributed to four key innovations, which introduce theoretical advancements at the image processing level while being specifically tailored to the characteristics of cultural tourism scenarios. These innovations fundamentally distinguish the proposed approach from existing methods. First, the multi-task collaborative interactive attention mechanism effectively eliminates the task separation between tourist behavior recognition and interest region generation. Through bidirectional information flow, mutual feature enhancement is achieved, in contrast to conventional approaches that rely on unidirectional attention guidance or lack inter-task interaction altogether. This design significantly improves feature utilization efficiency and constitutes a critical factor enabling simultaneous performance gains in both tasks. Second, the behavior-aware weighting mechanism integrates behavioral semantics and temporal information into interest region generation. By moving beyond traditional density-based paradigms, interest regions are generated in a manner that more accurately reflects tourists’ actual interests. This approach directly addresses the fundamental limitation of insufficient semantic correlation in existing methods. Third, the bidirectional guidance mechanism combining spatiotemporal graphs and region-level attention enhances semantic modeling capability in complex environments. Through refined graph structure design and attention-based feature fusion, both crowd interactions and temporal dynamics are accurately captured, thereby enabling effective adaptation to cultural tourism scenarios characterized by high crowd density and diverse behavioral patterns. Fourth, the fully differentiable end-to-end optimization framework ensures coordinated training across all modules. By eliminating non-differentiable components that typically introduce training discontinuities in conventional methods, improved convergence accuracy and generalization capability are achieved.

The unique characteristics of cultural tourism scenarios impose specific requirements on model design. High crowd density may lead to feature ambiguity, complex backgrounds can introduce substantial noise, and diverse behavioral patterns necessitate precise discrimination. These challenges are effectively addressed through the proposed framework by leveraging multi-scale feature extraction via the shared backbone network, scale adaptation through adaptive Gaussian kernels, and interaction modeling via multi-head attention mechanisms. As a result, strong adaptability to complex real-world scenarios is demonstrated.

The advantages of the proposed model are manifested in both theoretical contributions to image processing and practical value for cultural tourism applications. From the perspective of image processing theory, the fully differentiable design enables true end-to-end inference from pixel-level input to final output, ensuring seamless gradient backpropagation and providing a solid theoretical foundation for multi-task collaborative optimization. Furthermore, the proposed bidirectional attention mechanism introduces a novel paradigm for cross-task feature interaction, offering a new methodological perspective for multi-task collaborative inference in the field of image processing. From the perspective of practical application, the model demonstrates strong real-time performance and high accuracy, making it well-suited for real-time processing of surveillance video streams in digital cultural tourism platforms. Accurate recognition of tourist behaviors and precise identification of interest regions provide reliable support for scenic area management optimization, resource allocation, and service enhancement, thereby facilitating the digital and intelligent transformation of the cultural tourism industry. Despite these advantages, certain limitations are observed. In scenarios involving severe occlusion, the accuracy of pose keypoint estimation and appearance feature extraction is degraded, leading to a decline in behavior recognition performance. In addition, for small-scale tourist instances, behavioral features and spatial information are more susceptible to background interference, which reduces the precision of interest region generation. These limitations highlight areas requiring further improvement and underscore the objectivity and rigor of the study.

To address these limitations and align with emerging trends in image processing research, future work can be pursued along two main directions. From a technical perspective, the incorporation of Transformer-based architectures can be considered to enhance long-range temporal modeling, thereby improving the capture of behavioral dependencies and increasing robustness in occluded scenarios. In addition, multimodal information fusion techniques can be explored by integrating visual features with complementary modalities such as audio and environmental context, thereby enriching feature representation and mitigating interference caused by small-scale targets and complex backgrounds. From an application perspective, the generalizability of the proposed framework can be further investigated in other public environments, such as shopping malls, transportation hubs, and urban parks. Through appropriate parameter adaptation and scenario-specific optimization, the applicability of the framework can be extended. Moreover, the integration of edge computing techniques can be explored to enable model lightweighting and improve deployment efficiency on terminal devices. Such advancements would further enhance the practical value of the framework and promote the widespread adoption of multi-task image processing techniques in the intelligent analysis of public scenarios.