Fusing Diffusion Models with Structure-Aware Generation Networks for Environment Design Image Synthesis and Realism Enhancement

2.1 Framework overview

A latent diffusion model is adopted as the foundational generative backbone, and an integrated structure-aware realism-enhanced diffusion generation framework is constructed to address the prevalent issues of insufficient structural control precision, lack of physical realism, and design semantic drift in environment design image synthesis tasks. A schematic illustration of the overall architecture and full-chain optimization of StructDiff-Real is presented in Figure 1. The proposed framework relies on a complete technical pipeline encompassing conditional encoding, hierarchical feature injection, frequency-domain decoupled generation, and cross-modal alignment constraints, with systematic optimizations performed across three dimensions: model input, generative inference, and global constraints. An end-to-end generation system from raw design inputs to high-quality professional design image outputs is thereby established, which is capable of precisely accommodating the dual professional requirements of structural precision and visual realism in environment design scenarios.

1.png

Figure 1. Overall architecture and full-chain optimization schematic of StructDiff-Real

The core advantage of this framework lies in the deep synergistic coupling among its functional modules, rather than the mere superposition of independent functionalities. On the input side, the multi-level structural disassembly and encoding mechanism is designed to provide stable and reliable multi-scale structural anchor points for the generation process, thereby furnishing foundational geometric constraints for the subsequent frequency-domain decoupling optimization. On the generation side, the frequency-domain decoupled sampling and physical realism enhancement strategy is capable of optimizing textural details and physical lighting performance while preserving structural integrity, thereby eliminating generation defects arising from feature coupling. On the constraint side, the cross-modal retrieval-augmented mechanism, grounded in domain-specific professional priors, simultaneously calibrates structural morphology and design semantics, compensating for the incompleteness of abstract input information. Through feature propagation and joint loss constraints, the individual modules are integrated into an organic whole, fundamentally overcoming the adaptation deficiencies of general-purpose diffusion models in professional environment design generation scenarios.

To address the scale-adaptation limitations inherent in conventional unimodal feature injection approaches, a multi-scale structural hierarchical encoding and progressive feature injection mechanism is designed at the input side of the proposed framework, with the specific workflow illustrated in Figure 2. Through this mechanism, the underlying design sketch is decomposed along the spatial scale into three semantic hierarchies: global layout, intermediate morphological structures, and local details, which correspond respectively to macroscopic site massing, building facade setback profiles, and microscopic elements such as windows, doors, and landscape accessories. Three parallel branches are employed within the network to process information from each hierarchy independently, generating dedicated feature maps adapted to different receptive fields. In conjunction with the intrinsic denoising evolution patterns of the diffusion model, the feature injection process is governed by a time-step-dependent dynamic weight scheduling strategy. During the early denoising stage under high-noise conditions, the highest weight is assigned to global layout features to anchor the macroscopic composition. As the iterative process proceeds, the control emphasis is smoothly transferred to intermediate morphological features. In the final low-noise denoising stage, the injection intensity of local detailed features reaches its peak, serving to refine architectural components and environmental details. Through this time-varying fusion strategy—characterized by decay and enhancement over time steps—cross-scale feature interference is effectively decoupled, and rigorous hierarchical precision control is achieved along the spatial dimension.

2.png

Figure 2. Multi-scale structural hierarchical encoding and progressive feature injection mechanism

2.2 Structure-texture frequency-domain decoupled sampling strategy for realism enhancement

The denoising iterative process of existing latent diffusion models performs joint modeling of structural and textural features, with the optimization objectives of these two feature categories exhibiting inherent conflicts. Structural features prioritize global consistency in spatial layout and geometric topology, whereas textural features focus on local richness in surface details, illumination variations, and material expressions. A unified denoising constraint inevitably induces mutual interference between these features, rendering it difficult to simultaneously satisfy the dual professional requirements of structural precision and visual realism in environment design images. To address this problem, a two-dimensional discrete wavelet transform is introduced into the latent space of the diffusion sampling process, enabling complete decoupling of structural and textural features. For the latent variable z_t at any given sampling time step, frequency-domain decomposition is performed, yielding four orthogonal frequency components:

$\left\{z_t^{L L}, z_t^{L H}, z_t^{H L}, z_t^{H H}\right\}=\operatorname{DWT}\left(\mathrm{z}_t\right)$                    (1)

where, $z_t^{L L}$ denotes the low-frequency approximation component, which carries global spatial layout and geometric structural information and constitutes the core feature determining the morphological integrity of the design scheme; and $z_t^{L H}$, $z_t^{H L}$, and $z_t^{HH}$ are high-frequency detail components, corresponding respectively to edge textures and illumination details in the horizontal, vertical, and diagonal directions. Accordingly, the latent variable can be reconstructed into two independent representations—low-frequency structural components and high-frequency textural components—thereby enabling separate optimization of features with distinct attributes. Compared to decomposition in pixel space, the latent-space approach operates in a lower-dimensional space with reduced redundant information, effectively lowering computational overhead while aligning with the native latent-space modeling mechanism of diffusion models and circumventing feature degradation caused by repeated encoding and decoding operations.

Based on the frequency-domain decoupled feature representation system, a time-varying adaptive guidance weight mechanism is established, with differentiated denoising optimization strategies designed for structural and textural components, thereby constructing a progressive generation iterative logic. The overall denoising optimization formulation can be defined as:

$\begin{gathered}\epsilon_\theta\left(z_t, t, C\right)=\epsilon_\theta\left(z_t, t\right)+\gamma_{\text {struct }}(t) \cdot \nabla_{z_t^{L L}} \log p\left(z_t^{L L} \mid C\right) \\ +\gamma_{\text {texture }}(t) \cdot \nabla_{z_t^{\text {texture }}} \log p\left(z_t^{\text {texture }} \mid C\right)\end{gathered}$                (2)

where, $\epsilon_\theta$ denotes the noise residual predicted by the model, $C$ denotes the input structural constraint condition, and $\gamma_{\text {struct }}(t)$ and $\gamma_{\text {texture }}(t)$ denote the dynamic guidance weights for lowfrequency structural and high-frequency textural components, respectively. The weight parameters exhibit a linear monotonic variation pattern with respect to the diffusion time step $t$. During the early stage of denoising iteration, global structural composition is prioritized, with a high structural weight value employed to reinforce geometric constraints and suppress interference from high-frequency noise on contour formation. In the later iterative stage, the structural contours have largely stabilized, and the structural constraint weight is gradually reduced while the textural guidance weight is concurrently increased, enabling refined optimization of material details and lighting performance. The optimal scheduling intervals for the two weight sets are determined through a grid search method, achieving a dynamic balance between structural stability and textural richness and fundamentally eliminating generation defects arising from frequency-domain feature coupling at the generative mechanism level. The structure-texture frequency-domain decoupled sampling and physically aware discriminative constraint strategy are illustrated in Figure 3.

3.png

Figure 3. Structure-texture frequency-domain decoupled sampling and physically aware discriminative constraint strategy

To further enhance the physical plausibility of texture generation in alignment with optical and material principles governing real-world environmental scenes, a multi-dimensional physically aware discriminator is constructed to enable refined evaluative constraints on illumination consistency, shadow compliance, and material authenticity. The discriminator takes the generated image as input and outputs a four-dimensional quantitative scoring vector, thereby achieving comprehensive characterization of the physical attributes of the image:

$s=\Psi(\mathrm{x})=\left[s_{\text {light }}, s_{\text {shadow }}, s_{\text {material }}, s_{\text {global }}\right]^{\top}$                     (3)

where, $s_{\text {light}}$ characterizes the uniformity of global illumination direction, computed from the statistical variance of highlight features across multiple regions of the image; $s_{\text {shadow}}$ quantifies the degree of matching between shadow morphology and occluding object scales, reflecting the physical compliance of shadow generation; $s_{\text {material}}$ evaluates the naturalness of surface reflectance properties based on statistical features derived from the bidirectional reflectance distribution function; and $s_{\text {global}}$ comprehensively represents the overall visual texture quality of the image. The discriminator is trained through contrastive learning on real environment design images and model-synthesized images, with a gradient penalty mechanism incorporated to ensure training convergence stability. The overall loss function is formulated as:

$\begin{gathered}L_{\Psi}=E_{x_{\text {real }}}\left[\left\|\Psi\left(x_{\text {real }}\right)-1\right\|_2^2\right] +E_{\mathrm{x}_{\text {fake }}}\left[\left\|\Psi\left(x_{\text {fake }}\right)-0\right\|_2^2\right]+\beta \cdot\left\|\nabla_x \Psi(x)\right\|_2^2\end{gathered}$                 (4)

where, $x_{\text {real}}$ and $x_{\text {fake}}$ denote real samples and generated samples, respectively; 1 and 40 denote the ideal real-score and fake-score vectors, respectively; and $\beta$ denotes the gradient penalty coefficient, which is employed to constrain the magnitude of parameter updates, thereby preventing training gradient explosion and mode collapse, and ensuring stable convergence of the discriminator's evaluation accuracy across various physical attributes.

To effectively integrate physical prior knowledge into the diffusion sampling process, a gradient guidance mechanism is constructed based on the discriminator's realism scores, enabling refined correction of the latent variable denoising iteration direction. The optimized latent variable update rule is formulated as:

$z_{t-1}=z_{t-1}^{\text {base }}+\eta \cdot \nabla_{z_t}\left\|\Psi\left(D\left(z_t\right)\right)-1\right\|_2^2$              (5)

where, $z_{t-1}^{\text {base}}$ denotes the latent variable output by the standard diffusion sampling pipeline; $D$ denotes the pre-trained variational autoencoder decoder, which is responsible for mapping the latent variable to pixel space for realism evaluation; and $\eta$ denotes the realism guidance intensity coefficient. Considering that the overall image structure has not yet been fully formed during the early iterative stages, and premature introduction of physical constraints would interfere with the generation accuracy of spatial layout, the physical gradient guidance is activated only during the final thirty percent of the denoising process iterations. Through this temporal constraint strategy, precise optimization of physical properties—including textures, illumination, and materials—during the later stages is achieved while ensuring stable generation of multi-scale structures, thereby realizing synergistic optimization of structural fidelity and physical realism in environment design images, and comprehensively enhancing the professional applicability of the generated results.

2.3 Retrieval-augmented mechanism with cross-modal semantic-structure alignment

In environment design image generation scenarios, the input information constituted by hand-drawn sketches and brief textual descriptions is generally characterized by sparsity and ambiguity. Relying solely on basic conditional constraints is insufficient for precisely anchoring the designer's creative intentions, readily leading to semantic drift and disorganized design logic in the generated outcomes. To compensate for the informational deficiencies of abstract input conditions, the introduction of domain-specific professional prior knowledge constitutes an effective approach for enhancing the compliance of generated content. To this end, a standardized and structured environment design case knowledge base is constructed, providing reliable structural priors and stylistic textural priors for the diffusion generation process. The overall standardized definition of the knowledge base is formulated as:

$K=\left\{\left(I_i, S_i, T_i\right)\right\}_{i=1}^N$                 (6)

where, N denotes the total number of samples in the knowledge base; $I_i$ denotes a high-quality real environment design image covering mainstream design scenarios such as building facades, landscape sites, and streetscape interfaces; $S_i$ denotes the corresponding three-level hierarchical structural annotations, strictly adhering to the previously established structural partitioning rules of global layout, intermediate morphological structures, and local details, thereby ensuring consistency within the structural system; and $T_i$ denotes a standardized semantic tag system encompassing three core categories of semantic information: scene style, functional attributes, and material characteristics. All samples and their annotations have undergone dual verification by professional designers, with erroneous annotations and low-quality samples eliminated, thereby ensuring the precision and professionalism of the prior information contained in the knowledge base.

To overcome the spatial heterogeneity barriers between structural and semantic modalities, and to achieve collaborative retrieval and unified constraints across the two types of information, a dual-encoder cross-modal feature alignment architecture is established, enabling the embedding mapping of heterogeneous information into a common dimensional space. Through two independent encoders, the mixed information from structural conditions and semantic sketches is respectively parsed, and the two categories of features are mapped into a shared embedding space of uniform dimensionality. The feature encoding process can be formulated as:

$h_{\text {struct }}=E_{\text {struct }}(C), h_{\text {sem }}=E_{\text {sem }}(T$, Sketch $)$                (7)

where, $E_{\text {struct}}$ denotes the structural encoder, responsible for transforming the hierarchical structural conditions $C$ into high-dimensional structural features $h_{\text {struct}}$; and $E_{\text {sem}}$ denotes the cross-modal semantic encoder, which fuses textual semantics $T$ with visual features from hand-drawn sketches to output unified semantic features $h_{\text {sem}}$. The output dimensions of both feature types are unified to $d$. On this basis, information noise-contrastive estimation contrastive learning is adopted to achieve precise cross-modal feature alignment, whereby feature discrepancies between matched modalities are reduced through distance constraints on positive and negative sample pairs. The loss function is defined as:

$L_{\text {align }}=-\log \frac{\exp \left(\operatorname{sim}\left(h_{\text {struct }}, h_{\text {sem }}^{+}\right) / \tau\right)}{\sum_{j=1}^B \exp \left(\operatorname{sim}\left(h_{\text {struct }}, h_{\text {sem }}^j\right) / \tau\right)}$              (8)

where, sim(,) denotes the cosine similarity function; $\tau$ denotes the temperature coefficient, which is used to smooth the feature distribution and modulate gradient sensitivity; $B$ denotes the training batch size; $h_{s e m}^{+}$denotes the positive semantic feature that precisely matches the current structural features; and $h_{s e m}^j$ denotes the negative sample features within the same batch. Through iterative optimization of this loss, the structural and semantic modalities can be brought into deep spatial alignment, thereby providing foundational support for subsequent collaborative retrieval.

4.png

Figure 4. Dual-encoder cross-modal feature alignment and domain knowledge collaborative retrieval architecture

Based on the aligned shared embedding space, a dual-dimensional collaborative retrieval mechanism integrating structure and semantics is established, overcoming the one-sided deficiencies inherent in single-modality retrieval and achieving comprehensive and precise matching of reference samples. The dual-encoder cross-modal feature alignment and domain knowledge collaborative retrieval architecture are illustrated in Figure 4. During the inference stage, the similarity between structural features and semantic features is computed synchronously, and the comprehensive matching score is obtained through equal-weight fusion, from which the optimal set of reference samples best suited to the current design requirements is selected. The retrieval rule is formulated as:

$R=\underset{\left(I_i, S_i, T_i\right) \in K}{\operatorname{top}-K}\left[\begin{array}{l}\operatorname{sim}\left(E_{\text {struct }}(C), E_{\text {struct }}\left(S_i\right)\right) \\ +\operatorname{sim}\left(E_{\text {sem }}(T), E_{\text {sem }}\left(T_i\right)\right)\end{array}\right]$                   (9)

Through this mechanism, both spatial structural rationality and semantic stylistic consistency of the generated outcomes are simultaneously accounted for, effectively avoiding the style mismatch problems caused by structure-only retrieval as well as the layout disorganization issues induced by semantics-only retrieval, thereby providing high-quality and highly relevant domain prior samples for the generation process.

The retrieved reference samples are integrated into the diffusion generation pipeline through a dual-pathway approach, enabling full-chain optimization from both structural constraints and stylistic textural dimensions, thereby establishing a consistency constraint system that synergistically combines semantic and structural alignment. At the structural level, the hierarchical structural annotations of the reference samples are leveraged to supplement geometric anchor points, reinforcing the professional regularity of the generated layout. At the textural level, color statistics and texture primitive features are extracted from the reference images, and the high-frequency texture denoising process is optimized through adaptive instance normalization, thereby enhancing the domain-specific adaptability of material and lighting representations. A joint loss function that integrates semantic matching and structural alignment is constructed to perform global optimization on the generated results:

$L_{\text {sem-struct }}=\operatorname{CE}\left(\operatorname{Sem}\left(D\left(z_t\right)\right), T\right)+\lambda_{\text {ref }} \sum_{\left(I_r, S_r\right) \in R} \operatorname{dist}\left(\operatorname{Struct}\left(D\left(z_t\right)\right), S_r\right)$                     (10)

where, $C E$ denotes the cross-entropy loss, employed to quantify the degree of semantic matching between the generated image and the design text; $D$ denotes the variational autoencoder decoder, which performs the mapping from latent variables to pixel space; Sem( ) and Struct( ) denote the semantic classification operator and structural feature extraction operator, respectively; dist( , ) denotes the feature cosine distance function; and $\lambda_{\text {ref }}$ denotes the reference structural constraint weight. Through this loss, both semantic drift and structural distortion issues are simultaneously corrected, fully leveraging the prior constraint capacity of the domain knowledge base and significantly enhancing the stability and professional quality of generated outcomes in complex design scenarios.

2.4 Unified training objective and inference pipeline

The multiple optimization objectives—encompassing diffusion denoising, structural constraints, physical realism discrimination, cross-modal alignment, and retrieval-based semantic constraints—are integrated into a multi-task joint optimization loss function, enabling coordinated and balanced learning across all functional modules. The overall unified loss function is formulated as:

$L_{\text {total }}=L_{\text {diff }}+\lambda_1 L_{\text {struct }}+\lambda_2 L_{\Psi}+\lambda_3 L_{\text {align }}+\lambda_4 L_{\text {sem-struct }}$                        (11)

where, $L_{\text {diff}}$ denotes the foundational denoising loss of the latent diffusion model, which preserves the native image generation capacity of the model; $L_{\text {struct}}$ denotes the multiscale structural consistency loss, which constrains the spatial layout and geometric topological precision of the generated images; $L_{\Psi}$ denotes the physically aware discriminative loss, responsible for optimizing the physical plausibility of illumination, shadows, and materials; $L_{\text {align}}$ denotes the crossmodal feature alignment loss, which achieves feature-space unification between structural and semantic modalities; and $L_{\text {sem-struct}}$ denotes the semantic-structural collaborative constraint loss, which rectifies semantic drift in the generated outcomes. The coefficients $\lambda_1, \lambda_2, \lambda_3$, and $\lambda_4$ denote tunable balancing weights for each sub-loss term. The optimal weight configuration is determined through a grid search strategy on the validation set, effectively harmonizing the optimization gradients across different tasks, preventing any single task from dominating model training, and ensuring balanced improvement across all generative performance dimensions.

To address the issues of gradient competition, parameter oscillation, and convergence instability arising from multi-task joint training, a progressive three-stage training strategy is designed, in which parameter convergence and collaborative fine-tuning are performed module by module according to functional priority. During the initial training stage, all parameters of the pre-trained diffusion backbone are frozen, and only the multi-branch structure-aware control network and hierarchical adaptation weights are iteratively optimized. Relying on the foundational diffusion loss and structural constraint loss, prioritized convergence of structural control capability is achieved, ensuring that the model possesses stable multi-scale geometric constraint capacity. In the second stage, the backbone network and structural control module parameters are fixed, and the physically aware discriminator is independently trained until the loss stabilizes. The discriminator gradient guidance mechanism is then embedded into the sampling pipeline for fine-tuning, establishing the optimization capability for image physical realism. In the third stage, the parameters of the previously matured modules are locked, and cross-modal contrastive learning training of the dual encoders is performed, achieving precise alignment between structural and semantic features. Finally, a minor end-to-end fine-tuning of the global model parameters is conducted, enabling deep coupling among the three major modules—structural control, frequency-domain realism enhancement, and retrieval alignment—to accommodate the generation requirements of complex environment design images.

During the model inference stage, a complete and orderly high-fidelity image generation pipeline is formed through the collaborative interaction of the various modules. The input sketch, layout wireframe, and textual description are first preprocessed, from which standardized three-level hierarchical structural conditions and semantic representations are parsed. Feature embedding is then performed through the dual encoders, and the optimal reference samples are retrieved from the knowledge base. Throughout the diffusion sampling iterations, multi-scale structural features are integrated into the U-Net feature hierarchies according to the progressive injection rules, while frequency-domain decoupling optimization is simultaneously applied to the latent variables at each step, with low-frequency structural shaping and high-frequency textural refinement respectively accomplished through dynamic weight scheduling. In the later sampling stages, physically aware gradient constraints and retrieval-based prior alignment constraints are introduced to correct the denoising direction of the latent variables, further enhancing image physical plausibility and design semantic matching. Finally, latent variable decoding is performed through the pre-trained variational autoencoder, outputting environment design images that simultaneously exhibit structural precision, physical realism, and design semantic consistency.

The StructDiff-Real framework constructed in this study effectively addresses the issues of structural inaccuracy, insufficient realism, and semantic drift commonly encountered in conventional environment design image generation. However, certain technical limitations remain under extremely complex design scenarios and engineering deployment conditions. When confronted with complex architectural and landscape structures featuring curved irregular surfaces, multiply nested topologies, and ultra-high-density components, the fine-grained geometric constraint capability of the model falls short of achieving fully precise replication, leaving room for improvement in the structural reconstruction accuracy of extreme details. Furthermore, the generation performance of the retrieval-augmented mechanism is highly dependent on the sample coverage breadth and annotation precision of the domain case knowledge base. In low-sample scenarios—such as niche regional design styles and customized innovative designs—the constraint efficacy of prior information exhibits notable attenuation, thereby limiting the upper bound of the model's creative generation capacity. Additionally, although the latent-space frequency-domain decomposition operations and the gradient guidance mechanism of the physically aware discriminator enrich the generative constraint dimensions, they also increase the computational overhead of model inference to a certain extent, resulting in a modest inference latency compared to the base diffusion model, which renders it less suitable for the ultra-fast inference requirements of real-time interactive design rendering.

The core technical architecture of the proposed method possesses robust domain generalization capability and cross-task transfer potential, with each functional module not being confined to the single application scenario of outdoor environment design. The multi-scale structural hierarchical encoding, frequency-domain feature decoupling optimization, and cross-modal semantic alignment mechanisms introduced in this study can accommodate the general structural constraint requirements of spatial design tasks. Through merely updating the corresponding domain's structural annotation system and case knowledge base, seamless transfer can be achieved to related design domains—including interior spatial design and urban district planning—thereby enabling intelligent image synthesis across multiple scenarios. At the stylistic adaptation level, relying on the retrieval-augmented domain prior constraint system, the model adaptively accommodates the generation characteristics of diverse design styles—such as traditional Chinese, modern minimalist, and European classical—effectively balancing standardized design expression with differentiated stylistic creation, demonstrating excellent cross-style generation stability and providing a feasible technical pathway for the construction of general-purpose intelligent generation models for spatial design.

Targeting the industry pain points of low efficiency in traditional environment design visualization workflows and high costs of scheme iteration, this research has developed a practically deployable intelligent generation technology system with significant engineering application value and industrial empowerment potential. In practical design workflows, this method enables high-precision visual rendering of preliminary design drafts rapidly based on designers' lightweight sketch layouts and textual design descriptions, substantially shortening the turnaround time of conventional modeling and rendering processes. Concurrently, the model is capable of generating multiple structurally sound and stylistically differentiated scheme variants from a single core design logic, providing sufficient material support for designers to conduct scheme comparison and optimization, thereby effectively enhancing design iteration efficiency. This technology can be broadly applied to engineering scenarios such as architectural landscape scheme exploration, public space renovation design, and site planning visualization, assisting designers in completing preliminary creative realization and mid-stage scheme optimization, and promoting the digital transformation of the environment design industry toward intelligent and efficient paradigms.