Lightweight Deep Neural Network–Based Image Style Transfer for Pattern Design

Neural Style Transfer (NST)-the task of rendering a content image in the artistic style of a reference image-has evolved into a foundational capability at the intersection of computer vision, generative AI, and digital art. Since Gatys et al. [1] first demonstrated that deep convolutional neural networks (CNNs) can disentangle and recombine image content and style representations, the field has seen explosive growth spanning feed-forward networks [2], adaptive normalization [3], attention mechanisms [4, 5], Transformer architectures [6], and diffusion models [7, 8]; a comprehensive survey is provided by Jing et al. [9]. Industrial demand has grown commensurately, driven by large-scale commercial applications in high-end fabric printing, virtual garment rendering, UI/UX asset generation, and game content automation-all requiring high-fidelity, real-time, and spatially-controllable stylization.

Despite this progress, a critical performance gap persists when NST is deployed in precision pattern design rather than broad artistic filtering. Three fundamental tensions define this gap:

Tension 1: Naturalness vs. Structural Distortion. Industrial patterns demand material realism, stroke coherence, and pixel-accurate boundary preservation. Global statistical matching methods such as AdaIN [3] and WCT [10], which align feature mean/variance or full covariance statistics, inherently discard spatial topology. Under extreme style transformations, this produces fragmented patch-like artifacts at region boundaries — a failure mode visually unacceptable in textile and graphic design contexts. The root cause is that Gram matrix statistics [1] are position-agnostic aggregations: they capture what textures appear globally, but not where they are spatially consistent.

Tension 2: Local Controllability vs. Global Transfer. Professional pattern design imposes fine-grained spatial constraints: garment foregrounds and backgrounds must remain stylistically isolated, brand logo edges require sub-pixel smoothness, and repeating motifs must satisfy symmetry or periodicity constraints. Methods relying solely on global style loss optimization lack explicit local continuity guidance mechanisms and cannot meet these industrial-grade precision requirements.

Tension 3: Representational Capacity vs. Inference Efficiency. Transformer-based methods such as StyTr² [6] achieve superior generation quality through token-level cross-attention, but standard self-attention scales as $\mathcal{O}(\left( HW{{)}^{2}} \right)$ with image resolution, making 4K pattern printing computationally intractable and interactive design workflows (requiring sub-25 ms latency for 40+ fps) infeasible. Diffusion models [7, 8] push fidelity further but require 20–100 denoising steps, incurring latencies of seconds to minutes.

The core challenge is to find Pareto-optimal solutions in the three-dimensional objective space of {Expressiveness, Controllability, Inference Efficiency}. Existing methods tend to excel in at most two dimensions, sacrificing the third unacceptably. LiteStyleFusion addresses this through a principled decoupled three-component design: a lightweight CNN backbone for multi-scale feature extraction, a Low-Rank Tokenization Module for efficient attention computation, and a Pattern-Prior Constraint for spatially-controlled generation.

The main contributions of this paper are:

(1) LiteStyleFusion Framework. We propose a lightweight CNN-Transformer hybrid for real-time arbitrary style transfer targeting industrial pattern design. Through Learnable Low-Rank Tokenization, attention complexity is reduced from $\mathcal{O}(\left( HW{{)}^{2}} \right)$ to $\mathcal{O}\left( N_{t}^{2} \right)$ (${{N}_{t}}=256$vs. $HW=4096$ for $512\times 512$ inputs), achieving approximately 256× computational compression while maintaining state-of-the-art generation quality (FID = 23.5, SSIM = 0.756) at 19.6 ms inference latency on a single NVIDIA RTX 4090 GPU.

(2) Gated Multi-Head Cross-Attention (GMHCA). We introduce a novel attention aggregation mechanism that learns per-head scalar gate coefficients through back-propagation, conditioned on attention entropy statistics. This reduces FID by 1.6 and improves SSIM by 0.017 relative to standard equal-weight multi-head attention, with near-zero additional parameter overhead.

(3) Pattern-Prior Local Consistency Loss (${{\mathcal{L}}_{pattern}}$). We propose an explicit spatial constraint that enforces gradient continuity in shallow VGG feature space within edge-prior mask regions M (auto-generated via Canny detection [11] or user-specified). This mechanism fundamentally suppresses cross-boundary color bleeding artifacts, improving boundary sharpness scores by 18.2% in user evaluation ($p~<~0.01$).

(4) Systematic Evaluation. We provide a comprehensive experimental study encompassing: quantitative comparison with six baseline methods, structured ablation with 11 configuration variants, Pareto frontier analysis across the efficiency–quality space, a 60-participant perceptual user study with statistical hypothesis testing, and honest qualitative analysis including documented failure cases.

The remainder of this paper is organized as follows. Section II reviews related work on style transfer, efficient attention, and controllable generation. Section III details the LiteStyleFusion architecture, training protocol, and loss functions. Section IV presents experimental results, ablation studies, and user evaluation. Section V discusses limitations and future directions. Section VI concludes.

3.1 Dataset configuration and preprocessing protocol

Content domain: MS-COCO 2017 [23] provides over 120,000 natural scene images spanning diverse object categories, occlusion patterns, and multi-scale targets. We use 80,000 images for training and 5,000 validation images for quantitative evaluation.

Style domain: The WikiArt dataset [24] contains approximately 80,000 paintings from over 1,000 artists, spanning styles from classical realism to abstract geometrism across oil painting, watercolor, sketch, and printmaking media. As noted by Saleh and Elgammal [24], this collection provides sufficient diversity to evaluate arbitrary style generalization across both Western fine-art traditions and contemporary illustration styles. We use 70,000 images for training and 10,000 for evaluation.

Preprocessing: During training, content–style pairs are constructed via random sampling to encourage cross-style generalization. All images undergo: (1) proportional rescaling with short edge = 520 px; (2) random crop to 512 × 512; (3) random horizontal flip with p = 0.5; and (4) normalization using ImageNet statistics [25] (μ=[0.485, 0.456, 0.406], σ=[0.229, 0.224, 0.225]). At test time, images are rescaled and center-cropped to 512 × 512 to eliminate data augmentation variance. Table 2 summarizes the complete dataset split configuration used in all experiments.

Table 2. Dataset split configuration

3.2 Lightweight multi-scale feature encoder

We adopt MobileNetV3-Large [26] as the shared backbone encoder for both content image Ic and style image Is. MobileNetV3-Large [26] was selected over VGG-19 (used by StyTr² [6]) for three reasons: (i) 5.4M parameters vs. VGG-19's 143.7M represents a 26.6× parameter reduction; (ii) hardware-aware neural architecture search (NAS) optimizes it specifically for mobile NPU deployment; and (iii) its inverted residual blocks with hard-swish activation and depthwise separable convolutions provide 8–9× FLOPs savings over standard convolutions while maintaining competitive feature discriminability.

Feature extraction proceeds at three semantic levels: (a) shallow features ${{\Phi }_{\text{shallow}}}$ (Stage 3, stride 8, 40 channels) capturing local texture and edge gradients; (b) mid-level features ${{\Phi }_{\text{mid}}}$ (Stage 5, stride 16, 96 channels) encoding intermediate structural texture; and (c) deep features ${{\Phi }_{\text{deep}}}$ (Stage 7, stride 32, 960 channels) providing high-level semantic representations. For a 512 × 512 input, corresponding spatial resolutions are 64 × 64, 32 × 32, and 16 × 16.

A feature pyramid fusion module aggregates multi-scale information: ${{\Phi }_{\text{deep}}}$ is bilinearly upsampled $\times 2$ and channel-concatenated with ${{\Phi }_{\text{mid}}}$; the result is upsampled $\times 2$ and concatenated with ${{\Phi }_{\text{shallow}}}$. A $1\times 1$ convolution projects the combined feature to $\Phi \left( x \right)\in {{\mathbb{R}}^{64\times 64\times 256}}$, combining local detail from shallow layers with global semantics from deep layers.

3.3 Learnable Low-Rank Tokenization Module

Directly flattening $\Phi \left( x \right)\in {{\mathbb{R}}^{64\times 64\times 256}}$ into a token sequence would produce $N=4096$ tokens, resulting in per-head attention complexity $\mathcal{O}\left( {{N}^{2}} \right)=\mathcal{O}\left( {{4096}^{2}} \right)\approx 16.8M$ operations-computationally infeasible for batch real-time inference. The LRTM compresses dense spatial feature maps into compact token sequences of length ${{N}_{t}}$:

${{T}_{x}}=\mathcal{T}\left( \Phi \left( x \right);{{W}_{p}} \right)\in {{\mathbb{R}}^{{{N}_{t}}\times d}}$

where, ${{N}_{t}}=256$ is the number of sparse tokens and $d=512$ is the hidden dimension. The mapping T proceeds in two steps:

Step 1 — Depthwise Separable Convolution (DSConv): A 3 × 3 depthwise convolution followed by pointwise projection injects spatial inductive bias at approximately 1/9 the FLOPs of standard convolution, preserving local texture structure before spatial compression.

Step 2 — Adaptive Average Pooling (AAP): The convolved feature map is pooled to $\sqrt{{{N}_{t}}}\times \sqrt{{{N}_{t}}}=16\times 16$ spatial resolution, then flattened to an ${{N}_{t}}$-length token sequence. AAP is resolution-invariant and accommodates variable input sizes without retraining.

The resulting per-head attention complexity is $\mathcal{O}\left( N_{t}^{2} \right)=\mathcal{O}\left( {{256}^{2}} \right)=65536$ operations — a $256\times $ reduction over dense attention. This directly enables sub-20 ms single-image inference. The DSConv inductive bias prevents excessive information loss from abrupt spatial compression, a key advantage over simple random projection tokenization as used in Perceiver [17].

3.4 Gated Multi-Head Cross-Attention

Given content tokens ${{T}_{c}}\in {{\mathbb{R}}^{{{N}_{t}}\times d}}$ and style tokens ${{T}_{s}}\in {{\mathbb{R}}^{{{N}_{t}}\times d}}$, GMHCA establishes content–style co-representation using ${{T}_{c}}$ as queries and ${{T}_{s}}$ as keys and values:

$Q={{T}_{c}}{{W}_{q}},K={{T}_{s}}{{W}_{k}},V={{T}_{s}}{{W}_{v}}$

where, ${{W}_{q}},{{W}_{k}},{{W}_{v}}\in {{\mathbb{R}}^{d\times {{d}_{k}}}}$ are learnable projection matrices with ${{d}_{k}}~=~d/{{N}_{h}}$. Single-head attention output:

$\text{hea}{{\text{d}}_{h}}=\text{Softmax}\left( \frac{{{Q}_{h}}K_{h}^{\top }}{\sqrt{{{d}_{k}}}} \right){{V}_{h}}$

Standard multi-head attention concatenates all heads and applies a linear projection, implicitly assigning equal weight to each head. This is suboptimal in the style transfer context because: (i) cross-modal attention heads exhibit substantial heterogeneity — some learn focused, discriminative style–content correspondences while others produce diffuse, high-entropy distributions that degrade output quality; and (ii) the optimal head-weight distribution shifts with style complexity, making any static aggregation suboptimal. GMHCA addresses this with a learned dynamic gating mechanism:

${{T}_{\text{out}}}=\left( \mathop{\sum }_{h=1}^{{{N}_{h}}}{{g}_{h}}\cdot \text{hea}{{\text{d}}_{h}} \right){{W}_{o}}$

The scalar gate coefficient ${{g}_{h}}$ for each head $h$ is predicted by a lightweight two-layer MLP conditioned on the normalized attention entropy ${{H}_{h}}=-\mathop{\sum }_{i}{{p}_{i}}\log {{p}_{i}}$ of that head's softmax distribution. This gating design is conceptually related to channel attention mechanisms in discriminative networks [27], but operates at the attention-head level and is conditioned on distributional entropy rather than pooled feature statistics. Heads with low entropy (focused attention, high discriminative content) receive large ${{g}_{h}}$; heads with high entropy (diffuse attention, noise-prone) receive small ${{g}_{h}}$. All gate coefficients are learned end-to-end via back-propagation with no additional supervision signal. The MLP adds fewer than 100 parameters per head — negligible overhead. After fusion, ${{T}_{\text{out}}}$ is passed through an inverse tokenization decoder (transposed convolutions and bilinear upsampling) to recover the original spatial resolution, yielding ${{F}_{\text{fused}}}\in {{\mathbb{R}}^{H\times W\times C}}$, which is processed by a lightweight decoding head to produce ${{\hat{I}}_{\text{out}}}$.

3.5 Joint training loss

The model is trained end-to-end under empirical risk minimization with a four-term joint loss:

${{\mathcal{L}}_{\text{total}}}={{\lambda }_{c}}{{\mathcal{L}}_{c}}+{{\lambda }_{s}}{{\mathcal{L}}_{s}}+{{\lambda }_{\text{id}}}{{\mathcal{L}}_{\text{id}}}+{{\lambda }_{p}}{{\mathcal{L}}_{\text{pattern}}}$

Content Loss Lc:

Content preservation is measured via normalized MSE in mid-level semantic feature space (VGG-16 [12] relu3_2):

${{\mathcal{L}}_{c}}=\frac{\|{{\phi }_{\text{mid}}}\left( {{{\hat{I}}}_{\text{out}}} \right)-{{\phi }_{\text{mid}}}\left( {{I}_{c}} \right)\|_{2}^{2}}{{{C}_{\text{mid}}}\cdot {{H}_{\text{mid}}}\cdot {{W}_{\text{mid}}}}$

Style Loss Ls:

Following Gatys et al. [1], style is measured via Gram matrix discrepancy across VGG-16 layers $\left\{ relu{{1}_{1}},~relu{{2}_{1}},~relu{{3}_{1}},~relu{{4}_{1}} \right\}$:

${{\mathcal{L}}_{s}}=\mathop{\sum }_{l}{{\omega }_{l}}\|{{\mathcal{G}}_{l}}\left( {{{\hat{I}}}_{\text{out}}} \right)-{{\mathcal{G}}_{l}}\left( {{I}_{s}} \right)\|_{F}^{2}$

where, ${{\mathcal{G}}_{l}}=\phi _{l}^{\top }{{\phi }_{l}}/\left( {{C}_{l}}{{H}_{l}}{{W}_{l}} \right)$ is the normalized Gram matrix and layer weights ${{\omega }_{l}}=1/4$ (equal weighting across four layers, following standard practice [1]).

Identity Loss Lid:

When Ic = Is is sampled during training (20% of batches), an identity mapping penalty is applied:

${{\mathcal{L}}_{\text{id}}}=\|{{\hat{I}}_{\text{out}}}-{{I}_{c}}\|_{2}^{2}\left( \text{when }\!\!~\!\!\text{ }{{I}_{c}}={{I}_{s}} \right)$

This acts as an implicit regularizer, preventing style collapse (the tendency to generate a single dominant texture for all inputs) and improving generalization to unseen style–content combinations.

Pattern-Prior Local Consistency Loss ${{\mathcal{L}}_{pattern}}$:

Global statistical losses cannot prevent cross-boundary color bleeding — the primary artifact in pattern design applications. We introduce an explicit spatial constraint via edge-prior mask M (automatically generated by Canny [11] edge detection applied to IC, or user-specified interactively). The constraint enforces gradient continuity in shallow VGG feature space within mask regions M:

${{\mathcal{L}}_{\text{pattern}}}=\mathop{\sum }_{p\in \mathcal{M}}\|\nabla {{\phi }_{\text{shallow}}}\left( {{{\hat{I}}}_{\text{out}}} \right)\left( p \right)-\nabla {{\phi }_{\text{shallow}}}\left( {{I}_{c}} \right)\left( p \right)\|_{2}^{2}$

This constraint is applied only to ${{\phi }_{\text{shallow}}}$ (VGG $relu{{1}_{1}}$), which retains high-resolution spatial gradient information critical for boundary localization. Physically, it imposes gradient barriers at stylistically significant boundaries: the output's shallow feature gradients must match the content image's gradients at mask pixels, preventing style textures from bleeding across boundaries. The mask $\mathcal{M}$ is binary, with $\left| \mathcal{M} \right|/\left( HW \right)\approx 0.08$ on average for natural images, ensuring that the constraint is spatially targeted and does not globally suppress stylization freedom.

Loss Weights and Training Protocol:

Loss weights are determined by systematic ablation (Section IV-C): ${{\lambda }_{c}}=1.0$, ${{\lambda }_{s}}=10.0$, ${{\lambda }_{\text{id}}}=50.0$, ${{\lambda }_{p}}=0.5$. Training uses the Adam optimizer ($\text{lr}=1\times {{10}^{-4}}$, ${{\beta }_{1}}=0.9$, ${{\beta }_{2}}=0.999$, $\varepsilon ={{10}^{-8}}$) for 60 epochs with a cosine annealing schedule (lr decays to $\text{lr}/100$ over the final 10 epochs). Batch size=8, with 20% identity pairs. Training requires approximately 48 hours on a single NVIDIA RTX 4090 GPU (24 GB GDDR6X), CUDA 12.0, PyTorch 2.0.1.

LiteStyleFusion demonstrates that the efficiency–quality–controllability trilemma in style transfer can be substantially resolved through principled architectural co-design. Three key insights emerge from our analysis:

Insight 1: Spatial compression thresholds exist. The LRTM ablation establishes a practical rule: ${{N}_{t}}\approx HW/16$ (here $256=4096/16$) preserves perceptually sufficient style–content correspondence. Below this threshold, quality degrades precipitously; above it, gains are marginal. This finding may generalize to other cross-modal attention tasks on dense visual features.

Insight 2: Head heterogeneity is consequential. The GMHCA ablation demonstrates that 25%–50% of cross-attention heads in standard MHA consistently underperform (high entropy, low discriminative content). Entropy-conditioned gating provides a lightweight, unsupervised mechanism to suppress these heads dynamically, yielding significant quality gains at near-zero cost.

Insight 3: FID and SSIM measure orthogonal objectives. The λ_p sweep reveals a fundamental metric tension: ${{\mathcal{L}}_{pattern}}$ simultaneously improves SSIM (by preserving content structure) and reduces FID (by suppressing artifacts) up to ${{\lambda }_{p}}=0.5$, but beyond this point SSIM continues rising while FID degrades. This confirms that SSIM is a poor proxy for style transfer quality at high constraint strengths, and that FID-which measures distribution alignment with the target style-should be the primary evaluation criterion.

Limitations: (1) ${{\mathcal{L}}_{pattern}}$quality is bounded by mask accuracy. Complex scenes with dense occlusions (e.g., COCO categories 'crowd', 'pile') produce fragmented Canny masks, leading to incomplete boundary coverage and residual artifacts. An end-to-end learnable semantic edge predictor co-trained with the main framework would eliminate this external dependency, at the cost of additional training complexity. (2) WikiArt [24] coverage is geographically biased toward Western art traditions. East Asian styles (sumi-e, ukiyo-e), South Asian miniature painting, and Islamic geometric patterns are underrepresented, causing quality degradation for these domains. Training on culturally diverse multi-source datasets is a necessary extension for global deployment. (3) Per-frame independent stylization of video sequences produces temporal flickering due to the frame-to-frame inconsistency of stochastic stylization. Optical flow-guided temporal consistency constraints (e.g., penalizing deformation from the warped previous frame) are required for video extension. Without such constraints, flickering frequency scales approximately as $1/{{N}_{t}}$, making this limitation more pronounced at smaller token counts.

Future Directions: (1) End-to-end learnable mask generation for ${{\mathcal{L}}_{pattern}}$ via a lightweight semantic edge predictor co-trained with the main framework. (2) Training on culturally diverse multi-source datasets to improve global style coverage. (3) Video style transfer extension with optical flow-guided temporal coherence loss. (4) Multi-style regional blending: enabling simultaneous application of multiple style references to different spatial regions via learned multi-mask conditioning. (5) Mobile NPU deployment validation on Apple Neural Engine and Qualcomm Hexagon to verify real-world latency replicability of the 19.6 ms benchmark.