Generative Artificial Intelligence-Empowered Image Reconstruction and Visual Information Completion

2.1 Overall framework and feature encoding-decoding pipeline

The proposed structure-texture dual-branch collaborative generation network is built upon a latent diffusion model as its foundational architecture, through which the reconstruction and refinement of missing image information are accomplished in a low-dimensional latent space. By means of decoupled modeling of structural and textural features along with a collaborative fusion mechanism, the coupling between structural distortion and textural artifacts inherent in conventional generative methods is effectively resolved. An overview of the overall network architecture and the feature encoding-decoding pipeline of the structure-texture dual-branch collaborative generation network is presented in Figure 1. The inputs to the network are the original degraded image and a binary mask matrix, which are respectively defined as the image matrix $I \in \mathrm{R}^{H \times W \times 3}$ and the mask matrix $M \in\{0,1\}^{H \times W}$, where $H$ and $W$ denote the height and width of the image, respectively. To reduce the modeling overhead associated with high-resolution images, a variational autoencoder is employed for feature compression, with a downsampling factor set to 8 . The encoder maps the pixel-space image to a dimensionally compressed latent feature $z=\mathrm{E}(I) \in \mathrm{R}^{h \times w \times c}$, where $h=H / 8$ and $w=W / 8$ represent the spatial dimensions of the latent feature, and the channel dimension is $c=4$. Simultaneously, the original mask matrix is downsampled in a corresponding manner, yielding a mask feature $M_{\text {lat}}$ adapted to the latent space, thereby achieving precise alignment of missing-region information at the feature level.

tu_pian_1.png

Figure 1. Overall network architecture and feature encoding-decoding pipeline of the structure-texture dual-branch collaborative generation network

A dual-branch parallel encoding-decoding architecture is constructed within the network, through which global structural modeling and local texture generation are respectively accomplished, thereby enabling differentiated representation of the two categories of visual features. The structure branch adopts a four-layer symmetric encoder-decoder U-Net architecture, taking as input the combined features obtained by channel-wise concatenation of latent features and mask features, and is dedicated to extracting image edge contours, spatial layout, and geometric structure information, outputting structural features $F_s$ with dimensions of $h \times w \times 64$. During training, a Sobel gradient $L 1$ loss is imposed on this branch to constrain its learning, thereby reinforcing the model's capacity for capturing image structural features and ensuring the overall spatial plausibility of the reconstructed content. The texture branch incorporates cross-modal cross-attention units at the decoder stage, where query matrices are constructed from the branch's own textural features, while key and value matrices are constructed by reusing the structural features output from the structure branch. Through cross-layer feature interaction, global guidance of texture generation by structural priors is achieved, ultimately yielding refined textural features $F_t$ of the same dimensionality, ensuring that the texture generation process strictly conforms to the overall structural logic of the image.

To achieve adaptive fusion of structural and textural features, a spatial gated fusion module is introduced into the model, through which an adaptive weight distribution is learned via convolutional layers and a Sigmoid activation function, yielding a spatial gating matrix G. This computation is formulated as:

$G=\sigma\left(\operatorname{Conv}\left(\left[F_s ; F_t\right]\right)\right)$          (1)

where, $\sigma$ denotes the Sigmoid activation function, Conv denotes the convolution operation, and $[;]$ denotes the channel-wise concatenation operation. Adaptive fusion of the dual features is accomplished via the gating matrix, yielding the optimized latent feature:

$\hat{z}=G \odot F_s+(1-G) \odot F_t$           (2)

where, ⊙ denotes the element-wise multiplication operation. The fused latent feature is then fed into a three-layer Transformer decoder integrated with hybrid sparse selfattention for global feature refinement, through which feature noise and local deviations are corrected. Finally, feature upsampling and pixel-space reconstruction are performed via the variational autoencoder decoder, yielding the restored complete image $\hat{I}=D(\hat{z})$, through which bi-directional optimality of global structural regularity and local textural authenticity is achieved.

2.2 Context-adaptive dynamic diffusion sampling mechanism

An overview of the core innovative components and the multi-modal alignment micro-structure is presented in Figure 2. The latent diffusion model accomplishes the restoration of missing latent features through multiple rounds of reverse Gaussian denoising. The standard reverse generation process conforms to a Gaussian conditional distribution, mathematically formulated as:

$p_\theta\left(\mathrm{Z}_{t-1} \mid \mathrm{Z}_t\right)=N\left(\mathrm{Z}_{t-1} ; \mu_\theta\left(\mathrm{Z}_t, t\right), \Sigma_\theta\left(\mathrm{Z}_t, t\right)\right)$          (3)

where, $t$ denotes the current time step of the reverse diffusion iteration; $z_t$ denotes the latent feature map at step $t$; $\mu_\theta$ and $\Sigma_\theta$ respectively denote the distribution mean and variance predicted by the network; and $N$ denotes the multivariate Gaussian distribution. Conventional sampling strategies employ a fixed and uniform time step across the entire iterative cycle, rendering them incapable of adaptively distinguishing between smooth image regions and structural boundaries where abrupt transitions occur. The distributional deviation between known pixels and pixels to be generated at the missing-region boundaries cannot be progressively corrected throughout the iterative process, ultimately resulting in pronounced visual discontinuities between the restored region and the original image content. In this study, a context-adaptive discrepancy metric is designed, through which local distributional deviations are computed at each spatial location of the feature map during every denoising iteration, thereby guiding the dynamic adjustment of sampling step sizes and iteration counts, and embedding contextual consistency constraints throughout the complete diffusion generation pipeline.

tu_pian_2.png

Figure 2. Core innovative components and multi-modal alignment micro-structure of the proposed framework

For any spatial coordinate $p$ on the latent feature map, a $3 \times 3$ local neighborhood $N(p)$ centered at that coordinate is selected, with all pixels within the neighborhood collectively denoted as $q$. The Gaussian distribution parameters corresponding to intact pixels within the neighborhood are denoted as $\mu_{\text {known}}$ and $\Sigma_{\text {known}}$, while the distribution parameters corresponding to pixels to be generated in the missing region are denoted as $\mu_{\text {unknown}}$ and $\Sigma_{\text {unknown}}$. The mechanism simultaneously incorporates the Kullback-Leibler divergence and the mean squared deviation to quantify the degree of mismatch between the two classes of feature distributions. A weighted summation is performed to obtain the local discrepancy loss at a single pixel position at step t, with the complete formulation given as:

$\begin{gathered}L_{C A D}^{(t)}(p)=\sum_{q \in N(p)}\left[K L\left(N\left(\mu_{\text {known}}(q), \Sigma_{\text {known}}(q)\right)\right.\right. \\ \left.\left.\| N\left(\mu_{\text {unknown}}(q), \Sigma_{\text {unknown}}(q)\right)\right)+\lambda\left\|\mu_{\text {known}}(q)-\mu_{\text {unknown}}(q)\right\|_2^2\right]\end{gathered}$          (4)

where, $K L$ denotes the Kullback-Leibler divergence operator, used to quantify the overall distributional shape difference between the two Gaussian distributions; $\lambda$ denotes a hyperparameter for balancing the weights of the distributional shape constraint and the mean-distance constraint; and $\left\|\|_2^2\right.$ denotes the squared $\mathrm{L}_2$-norm operation. This loss function is computed synchronously with the diffusion time steps, enabling step-by-step and pixel-wise consistency supervision throughout the generation process, thereby avoiding the delayed correction issue associated with constraints imposed only at the final output stage.

Based on the local discrepancy loss, a spatially adaptive adjustment factor is constructed to determine whether refinement sampling iterations are required at a given position, formulated as:

$\alpha^{(t)}(p)=\sigma\left(\gamma \cdot L_{C A D}^{(t)}(p)-\beta\right)$           (5)

where, $\sigma$ denotes the Sigmoid activation function, which maps the computed result to the interval $[0,1] ; \gamma$ denotes a loss scaling coefficient, which amplifies the difference magnitude across different regions; and $\beta$ denotes a global bias threshold, which controls the trigger sensitivity of the refinement operation. A decision threshold of 0.5 is set in this study. When the adjustment factor exceeds this threshold, a significant feature distribution shift is indicated in the local region, necessitating additional refinement denoising. Conversely, when the value falls below the threshold, the region is determined to be texturally smooth, and the existing sampling precision is deemed sufficient for reconstruction requirements.

When a local region is determined to require refinement correction, a 5 × 5 effective range is demarcated centered at coordinate $p$, and refinement iterations are performed exclusively within this range. The refinement process employs a scaled time step $\eta \cdot \Delta t$, with the scaling coefficient fixed at 0.3 , while two additional independent denoising iterations are executed, with the total number of refinement iterations denoted as $N_{\text {refine}}=2$. The smaller iteration step size enables fine-grained adjustment of the latent feature distribution, and the two repeated iterations sufficiently smooth the distribution discontinuities between known and generated regions. For pixels outside the effective range, the original sampling step size is retained without additional computational overhead. In this manner, computational cost is incurred exclusively for local regions with complex structures and significant distribution deviations, while uniformly textural regions bypass redundant iterations, thereby controlling overall inference computational load while ensuring boundary naturalness.

2.3 Geometry-aware cross-modal structural guidance

The geometry-aware cross-modal structural guidance module is dedicated to injecting three-dimensional geometric priors into the image reconstruction process, thereby resolving the issues of perspective disorganization and occlusion logic contradictions inherent in two-dimensional pixel-driven generation. The core of this module lies in the deep synergy between a lightweight geometric estimation module and a cross-modal feature fusion mechanism. The module is composed of three components: geometric information generation, feature embedding, and multi-view constraints, through which precise geometric guidance is achieved without reliance on ground-truth three-dimensional annotations. The lightweight geometric estimation module adopts MobileNetV3 as its backbone network, complemented by a three-layer deconvolutional structure, forming an encoder-decoder architecture. Given the input image, the depth map and surface normal map are simultaneously estimated online, respectively denoted as $D \in \mathrm{R}^{H \times W}$ and $N \in \mathrm{R}^{H \times W \times 3}$, where H and W denote the height and width of the input image. The depth map records the relative spatial distance of each pixel, while the normal map describes the direction of the surface normal vector at each pixel.

The module is trained under a self-supervised learning paradigm, where a photometric loss function is constructed via cross-frame reprojection error, enabling the acquisition of geometric estimation capability without annotations. During training, adjacent frames in a video sequence are randomly selected as the source frame and the target frame. The source frame is projected into the target frame coordinate system based on the currently predicted depth and camera intrinsics, and the pixel-wise photometric error between the projected image and the target frame is computed. Through this error, the parameters of the geometric estimation module are optimized in a backward manner, circumventing reliance on large-scale annotated three-dimensional datasets. After geometric information generation, the concatenated features of the depth map and normal map are projected into a geometrically embedded vector $e_{\text {geo }} \in \mathrm{R}^{\mathrm{h} \times w \times 128}$ via a two-layer multi-layer perceptron, where h = H/8 and w = W/8 are consistent with the spatial dimensions of the latent feature map, and 128 denotes the channel dimension of the embedding vector, ensuring seamless integration of geometric information with subsequent generative features.

The geometric embedding is integrated into the generation network through a deep fusion approach, enabling geometric priors to guide the entire diffusion process. In each residual block of the U-Net, deformable convolution is first applied to the geometric embedding $e_{\text {geo}}$ for feature alignment, where pixel-level offsets are adaptively adjusted through learning to rectify spatial misalignment between geometric features and latent generative features, ensuring that geometric information precisely matches the structural distribution of the current generated region. The aligned geometric embedding and the current time-step embedding $e_t$ are added channel-wise and jointly fed into an adaptive group normalization layer, through which the intermediate features of the residual block are modulated via learned scaling factors and biases. This computational process is formulated as:

$F_{\text {mod}}=\gamma \cdot\left(F_{\text {res }}-\mu\right) / \sigma+\beta$           (6)

where, $F_{\text {res}}$ denotes the input feature of the residual block; $\mu$ and $\sigma$ denote the mean and standard deviation of the features within each group; and $\gamma$ and $\beta$ denote the adaptively learned parameters, which are generated by a fully connected layer from the fused features of the geometric embedding and the time-step embedding. In this manner, the geometric constraints are enabled to dynamically adapt to the feature distributions at different generation stages, achieving deep binding between geometric priors and the generative process.

For multi-view input scenarios, epipolar geometric constraints are further introduced into the module to reinforce the physical consistency of the three-dimensional structure, through which a composite geometric loss encompassing epipolar error and disparity consistency is constructed. First, the SuperPoint algorithm is employed to extract keypoints from the two-view images and perform matching, and the fundamental matrix $F \in \mathrm{R}^{3 \times 3}$ between the two views is solved via the eight-point algorithm. This matrix encodes the projective geometric relationship between the two views and satisfies the epipolar constraint $x_2^{\top} F x_1=0$, where $x_1$ and $x_2$ denote the homogeneous coordinates of corresponding points in the two views. Based on the fundamental matrix and the estimated depth, the geometric loss function is defined as:

$L_{\text {geo}}=\sum_i\left\|x_2^{(i) \top} F x_1^{(i)}\right\|_1+\left\|d_{\text {warp }}-d_{\text {epil }}\right\|_1$         (7)

where, the first term denotes the epipolar constraint loss, which iterates over all matched point pairs $\left(x_1^{(i)}, x_2^{(i)}\right)$ to compute the epipolar error, ensuring that corresponding points satisfy the geometric projection relationship; the second term denotes the disparity consistency loss, where $d_{\text {warp}}$ denotes the disparity map obtained by transforming the estimated depth map through camera parameters, and $d_{\text {epi}}$ denotes the disparity map derived from epipolar rectification. The reliability of depth estimation is guaranteed by enforcing consistency between these two disparity representations. This composite loss is jointly optimized with the reconstruction loss of the generation network, ensuring that the completed content is not only visually coherent in two-dimensional appearance but also physically consistent with the perspective and occlusion regularities of the real world in the three-dimensional geometric domain.

2.4 Sparse-activated hybrid attention and energy-gated feature alignment

To balance the capacity for long-range visual feature correlation modeling against computational resource overhead, while simultaneously alleviating the issue of attention weight distortion induced by image corruption masks, a sparse-activated hybrid attention mechanism is constructed in this study, through which decoupled modeling of spatial-dimension and channel-dimension correlations is achieved. This mechanism performs differentiated attention computation on input features, discarding the redundant computation paradigm of standard global dense attention in Transformers. For any given input latent feature X, the network first generates query, key, and value matrices via linear transformations, sequentially denoted as Q, K, and V. The total feature channels are equally divided into two independent branches, with the channel dimension of each branch taking half of the original dimensionality, respectively designated for spatial correlation modeling and channel correlation modeling, thereby enabling separated representation of the two types of feature dependencies and avoiding mutual interference between the two dimensions. An overview of the multi-scale progressive diffusion generation and dynamic sampling pipeline is presented in Figure 3.

tu_pian_3.png

Figure 3. Multi-scale progressive diffusion generation and dynamic sampling pipeline

In the spatial branch, a sparse attention selection strategy is adopted, through which effective correlative features in the image pixel space are focused upon, with the attention response matrix computed as:

$A=\frac{Q_s K_s^{\top}}{\sqrt{d / 2}}$            (8)

where, $Q_s$ and $K_s$ denote the query and key features of the spatial branch, respectively, and d/2 denotes the channel dimension of each branch, through which numerical normalization is achieved via the square root of the dimensionality, ensuring gradient propagation stability. A Top-K selection operation is performed on the response values of each row of the attention matrix, with the number of retained nodes set to $K=[N / 4]$, where $N$ denotes the total number of pixels in the feature space. After selection, only the top-ranked attention weights with the highest similarity are retained, while all remaining weights are uniformly set to negative infinity, followed by weight normalization via the Softmax function, yielding the refined spatial correlation features $Z_{\text {spatial}}$. In the channel branch, a grouped multi-head attention mode is adopted, where the query, key, and value features along the channel dimension are equally divided into 8 independent subgroups. Attention computation is performed independently within each subgroup, followed by feature concatenation, yielding channel-dimensional correlation features $Z_{\text {channel}}$. After channel-wise concatenation of the two sets of branch features, feature fusion is accomplished via linear projection, through which the computational complexity of attention is effectively reduced, optimizing the original standard attention complexity of $O\left(N^2 d_k+N d_k^2\right)$ to $O\left(K N d_k+N d_k^2 / G\right)$, thereby significantly reducing the inference overhead for high-resolution image reconstruction.

The energy-gated feature alignment module operates on the encoder-decoder skip connections of the network, through which adaptive screening and precise alignment of cross-level features are achieved, and representational noise introduced by ineffective features in missing regions is suppressed. In this module, the energy distribution of multi-level encoder features is computed to quantify the amount of effective information in the features, with the feature energy formulated as:

Energy$\left(F_{\text {enc }}^{(l)}\right)=\mid F_{\text {enc}}^{(l)} \|_2^2$          (9)

where, $F_{e n c}^{(l)}$ denotes the output feature of the encoder at the l-th layer, and the squared L₂-norm operation effectively characterizes the feature response intensity. The normalized feature energy is then fed into the Sigmoid activation function to generate adaptive feature gating weights:

Gate=Sigmoid$(E-\tau)$           (10)

where, E denotes the normalized feature energy matrix, and $\tau$ denotes the energy threshold parameter used to distinguish effective features from noise features. After the gating weights are applied to screen the encoder features, spatial position-adaptive alignment is accomplished via deformable convolution, through which spatial offset errors between encoder and decoder features are eliminated. Finally, cross-level feature fusion is performed, with the fusion rule satisfying:

$F_{\text {align}}^{(l)}=F_{\text {dec}}^{(l)}+$ Gate ⋅ Proj $\left(F_{\text {enc}}^{(l)}\right)$          (11)

where, $F^{(l)}{ }_{\text {dec}}$ denotes the decoder feature at the corresponding level, and Proj denotes the feature projection operation. This mechanism enables adaptive filtering of ineffective features in missing regions without reliance on mask information, substantially improving the feature fusion quality of skip connections.

The hybrid attention mechanism and the energy-gated alignment module form a complementary optimization framework. The former reduces computational redundancy and corrects attention biases induced by masks at the feature modeling level, while the latter purifies cross-level effective information at the feature fusion level. Through their synergistic operation, both the efficiency bottleneck and the noise interference issues of conventional Vision Transformers in image completion tasks are effectively addressed, while the fusion precision of global semantic features and local detail features is simultaneously reinforced, providing high-quality feature representations for subsequent refined image reconstruction.

2.5 Multi-scale progressive reconstruction strategy

To alleviate the difficulty in jointly optimizing global semantic layout and local high-frequency textures under the fixed-resolution reconstruction paradigm, a multi-scale progressive reconstruction strategy is constructed in this study, through which scene structure modeling and texture detail restoration are accomplished stepwise via a coarse-to-fine hierarchical iterative generation mechanism. The strategy first performs coarse-grained reconstruction in the low-dimensional space, where the original latent feature map is compressed to dimensions of $\mathrm{h} / 2 \times w / 2$, and low-frequency information restoration is accomplished via the DDIM sampler, with the total number of sampling steps set to $T_{coarse}=50$. A linear variance scheduling scheme is adopted during the sampling process to ensure the stability and continuity of the diffusion iterative process, with priority given to fitting the overall scene structure, object contours, and global semantic distribution of the image, thereby effectively circumventing the structural distortion and layout disorganization issues prone to occurrence in highdimensional iterative processes, and yielding a low-resolution latent feature $\hat{\mathrm{z}}_{low}$ with complete scene logic.

After global structural modeling is accomplished, texture detail optimization is achieved through progressive scale transformation and fine-grained iteration. A pixel shuffle algorithm combined with a 3 × 3 convolutional layer is adopted for feature upsampling, through which the feature discontinuities induced by scale transformation are smoothed while the high-resolution latent space dimensions are restored, ensuring the continuity of feature-space dimensionality switching. To activate the detail generation capability of high-dimensional features and enhance texture diversity, a small amount of Gaussian noise is added to the upsampled features, with the noise distribution satisfying:

$\epsilon \sim N\left(0,0.2^2 I\right)$          (12)

where, $N$ denotes the Gaussian distribution, 0.2 denotes the noise standard deviation, and $I$ denotes the identity matrix. The noise-added high-dimensional features are taken as the initial iterative features, and a refined reverse denoising process with $T_{\text {fine}}=10$ steps is performed. During the finegrained generation stage, the coarse-grained reconstructed features are incorporated into the generation constraints as additional cross-attention conditions, ensuring that the highfrequency texture generation process consistently adheres to the established global structure, thereby fundamentally suppressing local artifacts and textural disarray.

To further strengthen the feature constraint capability of multi-scale reconstruction, a multi-scale discriminator is introduced for comprehensive adversarial calibration of the hierarchical generation results. The multi-scale discriminator is composed of three structurally identical PatchGAN sub-networks, which perform feature discrimination on the original-resolution image, the 2× downsampled image, and the 4× downsampled image, respectively, covering multi-dimensional feature hierarchies including global scene, meso-scale structure, and micro-scale texture. An equal-weight fusion strategy is adopted for the adversarial losses across the three scales, yielding the final multi-scale adversarial loss function:

$L_{a d v}=\frac{1}{3}\left(L_{a d v}^0+L_{a d v}^2+L_{a d v}^4\right)$         (13)

where, $L_{a d v}^0, L_{a d v}^2$, and $L_{a d v}^4$ denote the adversarial losses at the original scale, the 2× downsampled scale, and the 4× downsampled scale, respectively. Through the multi-scale constraint mechanism, the global structural deviations at the coarse stage and the local textural errors at the fine stage are simultaneously calibrated, enabling synergistic optimization of global semantic integrity, structural geometric plausibility, and local textural authenticity.

2.6 Overall loss function and two-stage training strategy

A multi-objective composite loss function is adopted by the model to simultaneously constrain pixel-level reconstruction accuracy, contextual distribution consistency, three-dimensional geometric plausibility, generative distribution realism, and high-level visual perceptual features. The complete loss expression is formulated as:

$\begin{aligned} & L_{\text {total }}=\underbrace{\|I-\hat{I}\|_1+0.2\|\nabla I-\nabla \hat{I}\|_1}_{L_{\text {recon}}}+\lambda_1 L_{C A D}+\lambda_2 L_{\text {geo }} \\ & \quad+\lambda_3 L_{a d v}+\lambda_4 \underbrace{\sum_{j \in\{2,3,4\}}\left\|\phi_j(I)-\phi_j(\hat{I})\right\|_2}_{L_{\text {perc }}}\end{aligned}$           (14)

where, each sub-loss corresponds to a different level of generation constraint. $L_{\text {recon}}$ denotes the base reconstruction loss, composed of a weighted combination of pixel-space $L 1$ loss and gradient $L 1$ loss, with the gradient term weight set to 0.2 , through which edge and contour restoration effects are reinforced; $\lambda_1, \lambda_2, \lambda_3$, and $\lambda_4$ denote trainable balancing coefficients for adjusting the contribution of each auxiliary loss to the overall optimization objective; $L_{C A D}$ denotes the context-adaptive discrepancy loss, which enforces distribution matching between known and generated regions throughout the full diffusion iterative process; $L_{\text {geo}}$ denotes the epipolar geometric loss, ensuring that the reconstructed content conforms to the physical laws of three-dimensional perspective; $L_{\text {adv}}$ denotes the adversarial loss output by the multi-scale discriminator, optimizing the overall distributional realism of the generated images; and $L_{\text {perc }}$ denotes the perceptual loss, where $\phi_j$ refers to the pooled output features of the $2^{\text {nd}}, 3^{\text {rd}}$, and $4^{\text {th}}$ layers of the VGG-19 network, through which visual perceptual similarity is enhanced by reducing the high-level feature distance between the original image and the reconstructed image.

Direct end-to-end training of the multi-module coupled network is prone to gradient conflicts and localized module convergence lag. Therefore, a two-stage progressive training pipeline is designed in this study. In the first stage, a total of 50 epochs are executed, during which the backbone parameters of the diffusion model are completely frozen, and only the structure-texture dual branches, the lightweight geometric estimation module, and the energy-gated feature alignment module are updated. Priority is given to achieving parameter convergence of the sub-modules responsible for feature decoupling, three-dimensional prior embedding, and cross-level feature fusion, thereby simplifying the difficulty of solving the multi-task synchronous optimization problem. In the second stage, 30 epochs of global fine-tuning are performed, during which all network parameters are released for end-to-end joint optimization. The AdamW optimizer is adopted for training, with the weight decay fixed at 0.01. The learning rate follows a cosine decay scheduling rule, with the initial learning rate set to $1 \times 10^{-4}$, linearly decayed to $1 \times 10^{-6}$ at the end of the iteration cycle, through which smooth parameter adaptation across sub-modules is achieved, simultaneously enhancing image reconstruction accuracy and cross-scenario generalization performance.