Intelligent Garment Silhouette Design and Personalized Style Generation via Cross-Modal Visual-Semantic Alignment and Generative Image Processing

2.1 Overall framework overview

The proposed SilhouetteDiff framework is built upon Stable Diffusion v1.5 as the foundational generative model, with a latent-space generation system constructed via a frozen variational autoencoder and a trainable noise prediction network. The variational autoencoder is responsible for compressing pixel‑space images at 512 × 512 resolution into the latent space, with a fixed compression factor of 8 and an output channel dimension of 4, thereby substantially reducing computational complexity while preserving core structural and semantic information of the images. Departing from the pixel‑level single‑constraint paradigm commonly adopted in conventional garment generation models, the framework employs structurally explicit garment silhouette representation as the global anchoring element, establishing a progressive end‑to‑end generation architecture that sequentially accomplishes the full pipeline of semantic understanding, structural generation, personalized editing, and compositional creation. Three input modalities—textual descriptions, structural skeletons, and reference images—are integrated into the model, and high‑precision controllable generation is achieved through the coordinated coupling of four functional modules. A semantically aware cross‑modal alignment module for silhouette is designed to perform fine‑grained mapping of textual semantics, outputting semantic embedding features adapted to the domain‑specific characteristics of garments. A silhouette‑structured prior guidance module fuses structural keypoint coordinate information with semantic features, constraining the latent variable generation process through a dual‑path conditioning injection mechanism, thereby accomplishing precise mapping from semantic information to geometric structure. A silhouette‑preserving style editing module establishes a structured constraint mechanism based on contour masks, enabling contour fidelity during texture and color editing processes. A multi‑modal silhouette style disentanglement module performs disentangled modeling of latent‑space features, constructing mutually independent feature sub‑spaces for silhouette, texture, and color, thereby supporting flexible recombination of multi‑source design conditions and continuous style interpolation. All modules are deeply coupled through a unified silhouette representation serving as the shared feature carrier, forming a highly integrated and cohesive generation pipeline.

A two-stage progressive training strategy was adopted for model optimization, by which the gradient oscillation and parameter adaptation issues inherent in multi-module collaborative training were effectively resolved, and the dual objectives of preserving base model performance and achieving efficient convergence of newly added functional modules were simultaneously accomplished. In the first stage, all parameters of the variational autoencoder and the UNet backbone network were frozen, with training confined to the semantic alignment prototype matrix, adaptation layers, and structural control branch parameters, and preliminary optimization was performed via the base diffusion loss, cross-modal alignment loss, and text semantic loss. In the second stage, all newly added network parameters were unfrozen for end-to-end fine-tuning, with the overall joint loss function of the model defined as:

$L_{\text {total}}=L_{\text {simple}}+\lambda_1 L_{\text {align}}+\lambda_2 L_{\text {text}}+\lambda_3 L_{\text {sp}}+\lambda_4 L_{\text {orth}}+\lambda_5\left\|\Theta_{\text {new}}\right\|_2^2$             (1)

where, $L_{\text {simple}}$ denotes the base diffusion noise prediction loss, serving to preserve the fundamental image generation capability of the model; $L_{\text {align}}$ denotes the silhouette visualsemantic alignment loss, constraining the matching relationship between domain-specific fine-grained semantics and structural features; $L_{\text {text}}$ denotes the bidirectional text contrastive loss, reinforcing the robustness of textual semantic representations; $L_{s p}$ denotes the silhouette structure preservation loss, constraining contour integrity during editing processes; $L_{\text {orth}}$ denotes the subspace orthogonality loss, enabling multi-dimensional feature disentanglement; and $\Theta_{\text {new}}$ refers to all newly added trainable parameters of the framework, with $\lambda_1, \lambda_2, \lambda_3, \lambda_4$, and $\lambda_5$ serving as balancing weights for the respective loss terms. Model training was conducted using the AdamW optimizer, with first-order momentum coefficient set to 0.9 , second-order momentum coefficient to 0.999 , and weight decay coefficient to 0.01. In the first stage, the learning rate was set to $1 \times 10^{-4}$, with a batch size of 16 and training performed for 50 k steps; in the second stage, the learning rate was adjusted to $5 \times 10^{-5}$, with a batch size of 8 and training conducted for 100 k steps, augmented by a gradient accumulation step of 2 to mitigate hardware memory constraints. During the inference phase, the denoising diffusion implicit models sampling strategy was adopted, with a fixed sampling step of 50 and a classifier-free guidance scale of 7.5, thereby achieving an optimal balance between generation fidelity and inference efficiency.

2.2 Semantically aware silhouette cross-modal alignment module

To address the deficiency of fine-grained silhouette semantic representation in generic vision-language models when applied to the garment domain, a semantically aware silhouette cross-modal alignment module was designed, establishing a precise mapping mechanism between standardized silhouette semantics and visual geometric features. The architecture of the semantically aware silhouette cross-modal alignment module is illustrated in Figure 1. A categorical system was constructed by integrating five fundamental silhouette types from the garment design domain, and the textual description paradigm was constrained through a standardized prompt encoding strategy, with the diversity and generalizability of textual semantic representations being enriched by randomly incorporating ancillary attributes such as garment length and sleeve type. Feature extraction was performed via a pre-trained contrastive language–image pre-training encoder, yielding image visual features and textual semantic features, both uniformly represented in 768-dimensional space. To enforce clustering constraints on silhouette semantics across categories, a learnable silhouette prototype embedding matrix was introduced, with its parameters initialized via Kaiming uniform initialization and continuously updated and optimized throughout the training process, thereby generating category-specific semantic center representations and enabling structured modeling of professional garment silhouette semantics.

1.png

Figure 1. Architecture of the semantically aware silhouette cross-modal alignment module

A vision‑driven fine‑grained triplet loss function was constructed, in which an online hard example mining strategy was incorporated to enhance the model's discriminative capability for similar silhouette features, thereby effectively breaking the homogenization problem of closely related silhouette semantic representations within generic embedding spaces. With the visual geometric features of samples serving as the core constraint, this loss selects, within each training batch, cross‑category silhouette samples with the highest feature similarity as hard negative samples, and optimizes the discriminability of prototype representations through distance‑based constraints between positive and negative samples. The loss is formally defined as:

$L_{\text {align}}=\frac{1}{N} \sum_{i=1}^N \max \left(0,\left\|v_i-p_{y_i}\right\|_2^2-\left\|v_i-p_{y_j}\right\|_2^2+\alpha\right)$                 (2)

where, $N$ denotes the number of training samples in a single batch, $v_i$ denotes the visual features of the target sample, $p_{y_i}$ denotes the prototype embedding features corresponding to the true silhouette category of the sample, $p_{y_j}$ denotes the silhouette prototype features corresponding to the hard negative sample, and $\alpha$ denotes the margin threshold, with the optimal value determined as 0.5 via grid search on the validation set. In contrast to conventional text-based alignment approaches, this vision-driven constraint directly associates garment geometric structures with silhouette semantics, circumventing representational biases introduced by indirect textual mapping, and substantially improving the matching accuracy of fine-grained silhouette semantics.

To further refine the cross-modal alignment framework, a bidirectional text–prototype contrastive loss was constructed, enabling bidirectional adaptive adaptation between the textual modality and silhouette prototype features, thereby enhancing the robustness of semantic representations. The loss is expressed as:

$L_{\text {text}}=-\frac{1}{2 N} \sum_{i=1}^N\left[\begin{array}{c}\log \frac{\exp \left(\left\langle t_i, p_{y_i}\right\rangle / \tau\right)}{\sum_{k=1}^K \exp \left(\left\langle t_i, p_k\right\rangle / \tau\right)} \\ +\log \frac{\exp \left(\left\langle p_{y_i}, t_i\right\rangle / \tau\right)}{\sum_{j=1}^N \exp \left(\left\langle p_{y_i}, t_j\right\rangle / \tau\right)}\end{array}\right]$              (3)

where, $t_i$ denotes the textual semantic features corresponding to the sample, $K$ denotes the total number of silhouette categories, and $\tau$ denotes the temperature coefficient controlling the smoothness of feature distribution, with a fixed value of 0.07. This loss simultaneously enforces clustering constraints from text features toward their corresponding silhouette prototypes and matching constraints from prototype features toward their source texts, thereby achieving bidirectional deep alignment. On this basis, a lightweight adapter was configured and embedded into the backbone diffusion network, where feature optimization was performed through hierarchical transformations comprising layer normalization, dimensional compression, and dimensional restoration, with feature fusion accomplished via residual connections. The adapter was initialized with a differentiated parameter initialization strategy, by which the pre-trained generative capability of the base model was stably preserved. Deployed exclusively at mid-to-high resolution feature levels, the adapter achieves adaptive fusion of semantically aligned features with minimal parameter overhead, balancing model performance and inference efficiency.

2.3 Silhouette structure prior guidance module

To achieve quantitative modeling and precisely controllable generation of garment silhouette geometric structures, a structured prior guidance mechanism was constructed in this module, by which abstract garment appearance morphologies were transformed into a computable and constrainable geometric feature system, thereby addressing the issues of ambiguous structural constraints and poor proportional controllability in conventional generation methods. The workflow of the silhouette structure prior guidance module (SSPGM) is illustrated in Figure 2. A sparse semantic skeleton comprising ten coordinate points was defined, uniformly covering the core symmetric structures of the shoulder, chest, waist, hip, and hem, thereby fully characterizing the geometric proportions and overall morphology of garment silhouettes. Automatic keypoint detection was performed using a High-Resolution Network (HRNet-W48) pre-trained on the DeepFashion2 dataset, with redundant detection results filtered out via a non-maximum suppression algorithm, and all keypoint coordinates were uniformly normalized to the [0, 1] interval to eliminate scale discrepancies arising from varying image resolutions. For anomalous scenarios such as occlusion or detection failure, an interpolation completion strategy was designed based on the inherent geometric regularities of standard garment silhouettes, by which missing keypoint information was repaired through silhouette‑specific structural constraint rules, ensuring the geometric rationality and completeness of the skeletal features.

2.png

Figure 2. Workflow of the silhouette structure prior guidance module (SSPGM)

To transform the discrete sparse keypoint skeleton into dense features suitable for generative model input, two-dimensional Gaussian kernels were employed for keypoint heatmap rendering, by which continuous smooth structural condition feature maps were constructed. The feature computation at a single pixel position is defined as follows:

$G_m(p)=\exp \left(-\frac{\left|p-\left(x_m y_m\right)\right|_2^2}{2 \sigma^2}\right)$              (4)

where, p denotes the pixel coordinates in image space, ($x_m, y_m$) denotes the coordinates of the m-th normalized structural keypoint, and $\sigma$ denotes the Gaussian kernel scale parameter, which was set to 3.0 at the standard image resolution of 512 × 512, thereby generating an effective receptive field commensurate with garment structural scales. The Gaussian features corresponding to all keypoints were fused via a pixel-wise maximum strategy, yielding a single-channel global structural heatmap, which was subsequently expanded into a three-channel feature matrix $C_{\text {struct}} \in \mathrm{R}^{512 \times 512 \times 1}$ through channel replication, matching the input dimension requirements of ControlNet and accomplishing the standardized transformation from discrete geometric information to continuous structural features.

To extract the higher-order geometric ratio features implicit in the skeletal coordinates, a dedicated silhouette embedding encoding network was designed, enabling the mapping from low-dimensional coordinate data to high-dimensional structured semantic features. The overall encoding process is defined as:

$e_{\text {sil}}=M L P_{\text {sil}}(\operatorname{Flatten}(K))$                  (5)

where, Flatten(K) denotes the flattened skeletal feature vector with a dimension of 20, corresponding to the horizontal and vertical coordinates of the ten keypoints, and $e_{\text {sil}}$ denotes the 128‑dimensional silhouette structure embedding features output by the network. A progressive dimensional transformation structure of 20, 256, 128, and 128 was configured for the multi‑layer perceptron, with layer normalization and rectified linear unit activation functions applied to the first three layers, while only a linear transformation was retained in the final layer to fully preserve the higher‑order structural features. This encoding network was jointly and iteratively optimized with the model backbone network, enabling autonomous learning of the proportional relationships among core structures such as shoulder width, hem width, and waist position, thereby achieving an upgrade from basic coordinate information to refined silhouette structural representations.

A dual‑path conditioning injection mechanism was adopted in this module, integrating explicit geometric structural constraints and implicit semantic constraints to establish a dual‑layer synergistic regulation paradigm for generation. The first path is the ControlNet structural constraint path, which replicates the four‑stage downsampling encoding structure of UNet, with structural features being progressively injected through multi‑level zero‑convolution layers. The single‑stage feature fusion process is expressed as:

$f_{c t r l}^{(l)}=Z_{l, 3}\left(\operatorname{Block}_{c t r l}^{(l, 3)}\left(Z_{l, 2}\left(\operatorname{Block}_{c t r l}^{(l, 2)}\left(Z_{l, 1}\left(\operatorname{Block}_{c t r l}^{(l, 1)}\left(C_{\text {struct }}\right)\right)\right)\right)\right)\right)$              (6)

$f_{\text {UNet}}^{(l)}=f_{\text {UNet}}^{(l)}+f_{\text {ctrl}}^{(l)}$               (7)

where, $Z_{l, j}$ denotes the zero‑convolution layers, which employ 3 × 3 convolution kernels with all parameters initialized to zero, allowing progressive parameter updates during training and preventing initial constraints from interfering with the inherent feature distributions of the pre‑trained model; Block^(l,j)_ctrl denotes the ControlNet fundamental residual and Transformer modules, and f^(l)_ctrl outputs fine‑grained structural features at each stage, which are superimposed onto the UNet backbone features to achieve pixel‑level structural rectification. The second path is the semantic fusion path, in which deep coupling between silhouette embeddings and textual semantic features is accomplished through eight‑head cross‑attention. The fusion formula is defined as:

$c_{\text {joint}}=$ Concat$_{\mathrm{h}=1}^8\left[\operatorname{softmax}\left(\frac{\left(e_{\text {text }} W_Q^{(h)}\right)\left(e_{\text {sil }} W_K^{(h)}\right)^{\top}}{\sqrt{d_k}}\right)\left(e_{\text {sil }} W_V^{(h)}\right)\right] W_O$                 (8)

where, $e_{\text {text}}$ denotes the 768-dimensional contrastive language-image pre-training textual semantic features, and $W_Q, W_K$, and $W_V$ denote the attention query, key, and value projection matrices, respectively, $d_k$ denotes the single-head attention dimension with a value of 16, and $W_O$ denotes the feature dimension mapping matrix. The joint features obtained through fusion replace the original text conditions and are fed into the UNet cross-attention layers, enabling coordinated regulation of semantic information and geometric structure.

Model training and image inference were performed using the standard latent diffusion iterative mechanism. In the forward diffusion process, Gaussian noise was added to the clean latent variables, with the noise iteration formula defined as:

$z_t=\sqrt{\bar{\alpha}_t} z_0+\sqrt{1-\bar{\alpha}_t} \epsilon$                 (9)

where, $z_0$ denotes the original noise-free latent variable; $z_t$ denotes the noisy latent variable at step $t$; $\bar{\alpha}_t$ denotes the linear noise schedule coefficient; and $\epsilon$ denotes Gaussian noise following a standard normal distribution. With noise prediction accuracy serving as the optimization objective, the base diffusion loss function was defined as:

${L}_{\text {simple}}=E_{z_0, \epsilon, t, c_{\text {joint}}}\left[\left\|\epsilon-\epsilon_\theta\left(z_t, t, c_{\text {joint}}\right)\right\|_2^2\right]$             (10)

During the inference phase, the denoising diffusion implicit models deterministic sampling strategy was adopted, with a pure Gaussian noise latent variable $z_T \sim N({0, I})$ serving as the initial input, the sampling steps set to 50, and the deterministic sampling coefficient $\sigma_t=0$. The optimal latent variable $z_0$ was recovered through reverse iterative denoising, and was subsequently mapped to the pixel-space image $I_{\text {gen}}=D\left(z_0\right)$ via the variational autoencoder decoder $D$, yielding high-quality garment generation results with precise silhouette geometric structures.

2.4 Silhouette‑preserving style editing framework

A silhouette‑preserving style editing framework was constructed, in which a structure‑faithful texture update mechanism was established for personalized garment editing tasks, enabling semantic‑driven editing of fabric, color, and detail styles without altering the original garment silhouette structure. The schematic diagram of the silhouette‑preserving style editing framework is illustrated in Figure 3. The editing inputs to the framework are predefined as the original garment image and textual editing instructions, with the core optimization objective being to ensure that the edited image fully conforms to the textual semantic attributes while maintaining consistency in the geometric morphology of the garment outer contour. The framework first performs structured image decomposition and latent space inversion preprocessing. Garment semantic segmentation was accomplished via a DeepLabV3+ network fine‑tuned on the FashionGen dataset, from which the foreground mask of the image was precisely extracted. Outer contour extraction and hole filling were achieved through morphological closing operations with a kernel size set to 5 × 5, ultimately generating a silhouette mask with complete closed structure for subsequent global structural constraints. To align with the latent editing paradigm of diffusion models, the denoising diffusion implicit models inversion algorithm was employed to map the original image into the latent space, with the number of iterative steps set to 50, and a continuous latent variable trajectory was generated via the reverse update formula:

$\begin{gathered}z_{t+1}^{i n v}=\sqrt{\bar{\alpha}_{t+1}}\left(\frac{z_t^{i n v}-\sqrt{1-\bar{\alpha}_t} \epsilon_\theta\left(z_t^{i n v}, t, c_{\text {orig }}\right)}{\sqrt{\bar{\alpha}_t}}\right) +\sqrt{1-\bar{\alpha}_{t+1}} \epsilon_\theta\left(z_{t }^{i n v}, t, c_{\text {orig }}\right)\end{gathered}$          (11)

where, $z_t^{i n v}$ denotes the latent variable at the $t$-th reverse iteration step, $\bar{\alpha}_t$ denotes the linear noise schedule coefficient, $\epsilon_\theta$ denotes the noise prediction network, and $c_{\text {orig}}$ denotes the textual embedding features corresponding to the original image, encoded by the bootstrapping language-image pretraining model. The latent variable at the end of the iteration serves as the initial input for subsequent style editing.

3.png

Figure 3. Schematic diagram of the silhouette‑preserving style editing framework

During the latent-space editing stage, new textual condition features were generated based on the editing text instructions, and controllable updates of style textures were accomplished through a dual‑constraint mechanism. The editing instructions were fed into the contrastive language–image pre-training text encoder to obtain the editing condition features $c_{\text {edit}}$. At each denoising iteration step, a spatial mask constraint was introduced, by which the latent features of the background region were fixed, and texture updates were permitted only within the garment silhouette region. The latent variable update rule was defined as:

$\begin{gathered}z_t=z_t \odot \text {DownSample}\left(M_{\text {sil}}\right) +z_t^{\text {inv }} \odot\left(1-\text {DownSample}\left(M_{\text {sil}}\right)\right)\end{gathered}$            (12)

where, DownSample denotes the bilinear interpolation downsampling operation, by which the silhouette mask was scaled to a latent-space resolution of 64 × 64, and $M_{\text {sil}}$ denotes the silhouette mask of the original image. Concurrently, a classifier-free guidance strategy was introduced to enhance the style editing effect, with the optimized noise prediction expressed as:

$\tilde{\epsilon}_\theta=\epsilon_\theta\left(z_t, t, c_{\text {edit}}\right)+w \cdot\left(\epsilon_\theta\left(z_t, t, c_{\text {edit}}\right)-\epsilon_\theta\left(z_t, t, \varnothing\right)\right)$            (13)

where, $w$ denotes the guidance weight, fixed at 3.0, and $\varnothing$ denotes the empty text condition. Through the amplification of the difference between conditional and unconditional noise predictions, the guiding effect of textual editing semantics was enhanced, enabling precise updates of texture styles.

To thoroughly eliminate contour displacement and structural distortion arising during iterative editing, a multi‑dimensional joint silhouette preservation loss function was constructed, by which global rectification of the editing results was performed at both the pixel level and the edge structure level. The overall loss function is expressed as:

$\begin{gathered}L_{s p}=\operatorname{Dice}\left(\operatorname{SegNet}(\tilde{I}), M_{s i l}\right) +\lambda_1 \cdot {MS-SSIM}\left(\operatorname{Edge}(\tilde{I}), \operatorname{Edge}\left(I_{i n} \odot M_{s i l}\right)\right) +\lambda_2 \cdot\left(1-\operatorname{IoU}\left(\operatorname{SegNet}(\tilde{I}), M_{s i l}\right)\right)\end{gathered}$           (14)

where, Dice denotes the similarity coefficient loss, employed to constrain the pixel-wise overlap between the edited image and the original silhouette, with a tiny constant $\epsilon=10^{-6}$ introduced during computation to prevent division by zero; MS-SSIM denotes the multi-scale structural similarity loss, employed to constrain the morphological consistency of garment edge contours, with five scales configured in the model and corresponding weights of 0.0448, 0.2856, 0.3001, 0.2363, and 0.1333, respectively, and edge detection performed via the Canny algorithm with high and low thresholds set to 150 and 50, respectively; IoU denotes the intersection over union loss, serving as an auxiliary constraint to reinforce pixel-level matching. The weight coefficients $\lambda_1$ and $\lambda_2$ were set to 0.3 and 0.2, respectively, balancing structural edge constraints and regional matching constraints. During the inference phase, latent-space gradient optimization was performed based on this loss function, with the number of iterations set to 3 and the learning rate fixed at 0.05, through which silhouette deviations were further converged via reverse gradient correction.

This framework establishes a three‑level explicit shape‑preserving mechanism, comprising preprocessing mask constraints, iterative denoising spatial constraints, and post‑processing loss correction, which fundamentally differs from conventional methods that rely on implicit model regularization for structural maintenance during editing. Throughout the entire process, the standardized silhouette mask serves as the fixed constraint benchmark, exerting rigid control over the texture update process, thereby thoroughly circumventing silhouette deformation issues induced by texture transfer and semantic editing, and achieving a bidirectional balance between free garment style editing and absolute contour fidelity.

2.5 Multi‑modal silhouette style disentanglement generation strategy

To address the issue of garment silhouette, texture, and color features being highly coupled in the latent space and thus incapable of independent modulation, a multi‑modal silhouette style disentanglement generation strategy was designed, in which three mutually independent feature sub‑spaces were constructed to enable the decomposed representation and free recombination of core garment visual attributes. The architecture of the multi‑modal silhouette style disentanglement generation strategy is illustrated in Figure 4. This strategy relies on three dedicated encoders for disentangled feature extraction, corresponding to silhouette structure, fabric texture, and color attributes, respectively. The silhouette encoder adopts the structured multi‑layer perceptron architecture, with two additional geometric ratio features—shoulder‑to‑hem width ratio and waist‑to‑hip ratio—incorporated on the basis of the original 20‑dimensional keypoint coordinates, yielding a 22‑dimensional input vector, from which a 128‑dimensional silhouette representation is output through a dimensional transformation network, thereby enhancing the model's perception of global geometric morphology. The texture encoder takes the pure texture image within the garment mask region as input, with frozen parameters of an ImageNet‑pre‑trained ResNet‑50 employed to extract base texture features, and a two‑layer projection network used to map the 2048‑dimensional high‑dimensional features to a 128‑dimensional unified representation space, with only the projection layer parameters being fine‑tuned to preserve general texture priors. The color encoder computes the Red–Green–Blue (RGB) mean, standard deviation, skewness, and kurtosis of the effective garment region, which are concatenated into a 12‑dimensional color statistical vector, from which a 128‑dimensional color representation is output through a lightweight mapping network, precisely quantifying the overall color distribution characteristics of the garment. The output dimensions of the three encoders are unified, providing a standardized representation foundation for subsequent sub‑space orthogonal disentanglement and feature fusion.

To achieve complete disentanglement across different feature dimensions, an enhanced subspace orthogonality loss function was constructed, constraining the feature distribution from two perspectives—cross‑subspace independence and intra‑subspace dimensional independence—thereby thoroughly eliminating feature redundancy and coupling interference. The loss function is defined as:

$\begin{gathered}L_{\text {orth}}=\left\|Z_{\text {sil}}^{\top} Z_{\text {tex}}\right\|_F^2+\left\|Z_{\text {sil}}^{\top} Z_{\text {col}}\right\|_F^2 +\left\|Z_{\text {tex}}^{\top} Z_{\text {col}}\right\|_F^2+\beta \sum_{k\in\{\text {sil,tex,col\} }\}}\left\|Z_k^{\top} Z_k-I\right\|_F^2\end{gathered}$                (15)

where, $Z_{\text {sil}}, Z_{\text {tex}}$, and $Z_{\text {col}}$ denote the feature matrices of silhouette, texture, and color within a batch, respectively, with matrix dimensions of batch size multiplied by 128, and each row corresponding to the feature vector of a single sample; $\left\|\|_F^2\right.$ denotes the squared Frobenius norm, employed to measure the overall correlation between matrices; the first three terms constrain the cross-correlation between any two feature sub-spaces, forcing features of different attributes to be mutually independent; the final term serves as the subspace orthogonality regularization term, constraining the linear independence of dimensions within each individual feature space, with the parameter $\beta$ set to 0.1 to balance the constraint strengths between cross-space disentanglement and intra-space regularization. This loss effectively decouples the implicit correlations among the three visual attributes, constructing statistically fully orthogonal feature sub-spaces.

During the model inference phase, the framework achieves compositional generation from cross‑source conditions based on the disentangled sub‑spaces, supporting free combination of silhouette, texture, and color features. The independently obtained 128‑dimensional silhouette, texture, and color representations were concatenated into a 384‑dimensional combined conditional feature, which replaced the traditional single text condition as input to the diffusion model. To accommodate the multi‑condition fusion requirement, the UNet cross‑attention module was reconstructed, with a three‑branch independent conditional attention mechanism designed, in which dedicated projection matrices were configured for each of the three feature types to accomplish independent mapping of query, key, and value vectors. The attention features from each branch were adaptively fused via learnable gating weights, with the fusion formula defined as follows:

$f_{\text {out}}=\sum_{m \in\{\text {sil,tex,col}\}} \gamma_m \cdot \operatorname{softmax}\left(\frac{Q K_m^{\top}}{\sqrt{d_k}}\right) V_m$              (16)

where, $Q$ denotes the query vector obtained from UNet image feature mapping, $K_m$ and $V_m$ denote the key and value vectors corresponding to the three disentangled feature types, respectively, $d_k$ denotes the single-head attention dimension, and $\gamma_m$ denotes the learnable gating weights, with initial mean allocation set to one-third each, which can be adaptively adjusted during training to regulate the generation weights of each visual attribute, achieving fine-grained collaborative regulation of multiple conditions.

Leveraging the linear properties of the orthogonal disentangled sub‑spaces, this strategy enables continuous smooth interpolation generation of garment silhouette styles. Linear weighted fusion was performed on two different silhouette representations within the silhouette feature sub‑space, with the interpolation formula defined as:

$z_{s i l}^{(\alpha)}=(1-\alpha) \cdot z_{s i l}^{(A)}+\alpha \cdot z_{s i l}^{(B)}$                (17)

where, $\alpha$ denotes the interpolation coefficient, with a value range of 0 to 1, through which a gradual transition between the two silhouette morphologies is achieved via uniform sampling. Owing to the strict orthogonal constraints imposed on the sub-spaces, the feature space possesses a stable linear structure, and the interpolated intermediate samples can maintain semantic continuity and structural integrity without morphological distortion or feature entanglement, thereby enabling the generation of rich transitional silhouette styles and substantially expanding the creative space for personalized garment design.

4.png

Figure 4. Architecture of the multi‑modal silhouette style disentanglement generation strategy

2.6 Joint training and loss balancing

All functional modules in this study share a unified diffusion generation backbone. To achieve multi‑task collaborative optimization and address the issues of gradient imbalance and convergence instability arising from the superposition of multiple constraint losses, a multi‑dimensional integrated global loss function was constructed, enabling joint iterative optimization of model parameters. The global loss function integrates the base diffusion generation constraint, cross‑modal semantic alignment constraint, silhouette structural fidelity constraint, and feature disentanglement constraint, while a regularization mechanism is concurrently introduced to suppress overfitting risks in the newly added modules. The specific expression is as follows:

$L_{\text {total}}=L_{\text {simple}}+\lambda_1 L_{\text {align}}+\lambda_2 L_{\text {text}}+\lambda_3 L_{\text {sp}}+\lambda_4 L_{\text {orth}}+\lambda_5\left\|\Theta_{\text {new}}\right\|_2^2$                 (18)

where, $L_{\text {simple}}$ denotes the standard diffusion noise prediction loss, responsible for maintaining the base visual quality and distributional plausibility of image generation. $L_{\text {align}}$ and $L_{\text {text}}$ denote the silhouette semantic alignment loss and the textual semantic matching loss, respectively, employed to constrain the semantic consistency of cross-modal features. $L_{s p}$ denotes the silhouette preservation loss, employed to ensure the structural integrity of garment contour during style editing. $L_{\text {orth}}$ denotes the subspace orthogonality loss, achieving disentangled representation of silhouette, texture, and color features. The final regularization term is the L2 weight decay constraint, applied exclusively to the newly added network parameters $\Theta_{\text {new}}$, with the hyperparameter $\lambda_5$ fixed at $10^{-5}$, through which model generalization capability is enhanced by constraining parameter magnitudes.

To balance the constraint weights of different loss terms and accommodate the optimization priorities of each sub-task, a hierarchical grid search strategy was employed for hyperparameter calibration. Given that the numerical magnitudes of the semantic alignment loss and the text matching loss were similar, the weight parameters $\lambda_1$ and $\lambda_2$ were fixed at 0.1. Under this premise, a two-dimensional grid search was conducted for the structure preservation weight $\lambda_3$ and the feature disentanglement weight $\lambda_4$, with the search range covering 0.01, 0.05, 0.1, and 0.2. The harmonic mean of silhouette structural similarity and silhouette classification accuracy on the validation set was adopted as the optimization evaluation metric, and the optimal weight combination was ultimately determined as $\lambda_3=0.05$ and $\lambda_4=0.05$. Model training was divided into two progressive stages. In the first stage, only the base diffusion loss and the semantic constraint losses were enabled, prioritizing the convergence of the model's fundamental generation capability and cross-modal alignment capability. In the second stage, all loss terms were introduced to achieve full-module collaborative optimization. Additionally, a gradient clipping threshold of 1.0 was set during training, effectively avoiding the gradient explosion problem in multi-loss joint optimization scenarios and ensuring stable convergence throughout the overall training process.

The SilhouetteDiff framework constructed in this study, through structured silhouette prior constraints and multi‑modal disentangled representation learning, effectively addresses the challenges of fuzzy semantic alignment, unstable structural control, and silhouette distortion during editing in conventional garment generation models. The designed multi‑module collaborative architecture achieves integrated modeling of garment silhouette semantic understanding, geometric structure generation, personalized style editing, and multi‑attribute compositional creation, demonstrating superior generation precision and robustness in standardized flat garment design tasks. The experimental results fully validate the core role of structured silhouette anchors in enhancing garment generation controllability, demonstrating that the integration of geometric structure priors with cross‑modal semantic fusion constitutes an effective technical pathway for achieving fine‑grained intelligent garment design. Despite the significant performance advantages of the proposed method, certain limitations remain in terms of adaptation to complex real‑world scenarios and model generalization capability.

First, the structural representation capability of the proposed method relies on the prediction accuracy of the keypoint detection network. Under complex scenarios involving dense garment folds or external occlusions, keypoint localization is prone to deviation, leading to distortion in the subsequent silhouette constraint conditions and ultimately compromising the structural accuracy of the generation results. Second, this study only models and generates five fundamental standard silhouette types, with the constructed representation system being relatively coarse‑grained, incapable of precisely characterizing various gradient and fine‑tuned sub‑silhouette morphologies, and thus inadequately equipped for high‑precision, diversified garment design requirements. Finally, the training data predominantly consists of two‑dimensional flat garment samples captured from a frontal view, with a limited data scenario that does not cover garment deformation features corresponding to multiple human poses and body types, rendering the model insufficiently adaptable to personalized silhouette generation tasks in real wearing scenarios.

Addressing the limitations of the current study, future work will be continuously advanced along three directions: model robustness, representation refinement, and scenario expansion. The serial architecture relying on an independent detection network will be abandoned, and an end‑to‑end structure prediction mechanism will be constructed by leveraging the native visual perception capability of the diffusion model, thereby circumventing the error accumulation introduced by external detection models and enhancing the stability of skeleton representation under complex scenarios. Concurrently, a hierarchical fine‑grained silhouette label system will be established to enrich the semantic categories of silhouettes, and the zero‑shot learning paradigm will be incorporated to extend the model's generalization capability, enabling adaptive generation of unseen silhouette styles. Furthermore, the research boundary beyond two‑dimensional image generation will be transcended, with the structured silhouette prior mechanism being extended to the domain of three‑dimensional garment modeling, enabling multi‑view consistent three-dimensional silhouette generation, thereby further advancing intelligent garment design technology from flat simulation toward deployment in real three‑dimensional application scenarios.