Comparative Evaluation of CycleGAN Variants for Neural Style Transfer: Translating Floral Images into Madurese Batik Motifs

This study explores three CycleGAN versions: CycleGAN, ESA-CycleGAN, and AdaIN-CycleGAN. Each model translates floral images into patterns that resemble Madurese batik. The overall workflow of the experiment follows a simple sequence of steps. First, the dataset is assembled, and then the images are processed and normalized. Next, each model applies its own transformation method, and the outputs are evaluated using PSNR and LPIPS. Although the same images are used for all three models, each architecture handles style information differently, leading to distinct visual outcomes. Figure 1 provides an overview of this process.

Figure 1. Proposed method for neural style transfer (NST) from flora images to Madurese batik

2.1 Dataset description and preprocessing

The dataset contains two unpaired image domains (Figure 2): floral photographs in Domain A and Madurese batik motifs in Domain B. We prepared 360 training images for each domain, along with five additional images for testing. One example from Domain A is a photograph of an orchid with petals showing a gradual color transition from pink to purple. In this image, the variation in the red and green channels is quite noticeable; the texture of the petals stays distinguishable, too. For Domain B, we use a Madurese batik motif with floral elements. The dominant colors are white, black, and maroon, combined with ‘isen’ pattern and bold outlines that are characteristic of beach-inspired Madurese batik. The corresponding histogram shows strong color contrasts, especially in the blue, red, and green channels.

Figure 2. Sample images and Red–Green–Blue (RGB) histograms from the flora (Domain X) and batik (Domain Y) datasets

We adjusted all images to a size of 256 by 256 pixels. Then converted them into Red–Green–Blue (RGB) format and scaled them to a value range of -1 to 1. No data augmentation was applied, so every image used in the experiment was exactly as collected. This preprocessing was done to keep the input format uniform across the models, which helps when comparing how each one translates the floral images into Madurese batik motifs.

2.2 Model-specific transformation modules

Each architecture model introduces different modules for handling style and content features.

2.2.1 Cycle-Consistent Generative Adversarial Network implementation

CycleGAN is a Generative Adversarial Network class designed for image-to-image translation tasks with unpaired data. Unlike conventional methods, such as Pix2Pix, which require paired input and output images, CycleGAN maps two domains separately by applying the cycle consistency theory. Its architecture consists of two generators, G and F, and two discriminators, D_X and D_Y. Generator G translates images in domain X (e.g., flower images) to domain Y (e.g., Madura batik design images), and generator F translates images in domain Y to domain X.

In both domains, both discriminators distinguish between original and generated images. Through the collaboration of the generators and discriminators, CycleGAN produces realistic target images with the same organizational structure as the original images, without using paired data. For instance, a flower image can be transformed into a Madura batik pattern, as shown in Figure 3, using data from both domains. To generate realistic images while preserving the original content structure, CycleGAN applies three loss functions: adversarial, cycle-consistency, and total [25].

Figure 3. Cycle-Consistent Generative Adversarial Network (CycleGAN) flora and batik Madurese

1. Adversarial loss: It ensures that the discriminator cannot differentiate between the generated and original images from the generator. It is used to make the transformation results generated in the target domain look realistic. The adversarial loss function is shown in Eq. (1).

$\begin{aligned} & L_{G A N}\left(G, D_Y, X, Y\right)=\mathbb{E}_{y \sim P \text {data}(y)}\left[\log D_Y\right] \left.+\mathbb{E}_{x \sim P \text {data}(x)}\left[\left(1-\log D_Y G(x)\right)\right)\right]\end{aligned}$      (1)

2. Cycle-consistency loss: This loss function ensures that the transformed images can return to their original form with minimal difference. Supposing a flora image is first changed into a batik pattern and then changed back, the final image should look quite similar to the original. Eq. (2) computes the cycle-consistency loss.

$\begin{aligned} L_{c y c}(G, F) & =\mathbb{E}_{x \sim P \text {data}(x)}\left[\|F(G(x))-x\|_1\right]  +\mathbb{E}_{y \sim P \text {data}(y)}\left[\|G(f(y))-y\|_1\right]\end{aligned}$      (2)

3. The total loss function combines the two losses above with a balance parameter λ, which determines how strongly the model maintains cycle consistency compared with its ability to produce realistic images. The total loss function is computed using Eq. (3).

$\begin{aligned} L\left(F, D_X, D_Y,\right) & =L_{G A N}\left(G, D_Y, X, Y\right) +\left(F, D_X, Y, X\right)+\lambda L_{c v c}(G, F)\end{aligned}$      (3)

CycleGAN employs a ResNet-based generator with nine residual blocks arranged in an encoder–residual block–decoder structure (Table 1). This design allows the model to maintain the floral image structure while applying the batik style. The discriminator uses a 70 × 70 PatchGAN (shown in Table 2), which examines small patches of the image to ensure that the generated batik textures appear detailed and realistic, similar to traditional batik.

Table 1. Cycle-Consistent Generative Adversarial Network (CycleGAN) generator architecture used for flora-to-batik style translation

Table 2. CycleGAN and Adaptive Instance Normalization CycleGAN (AdaIN-CycleGAN) discriminator architectures used for flora-to-batik style translation

The model is optimized using least squares GAN loss, cycle-consistency loss to maintain structural fidelity, and an optional identity loss to preserve color statistics when necessary. Training is conducted for 50 epochs with a batch size of 1 using the Adam optimizer (learning rate 2 × 10⁻⁴, β₁ = 0.5, β₂ = 0.999) and employs loss weights λ_cycle = 10 and λ_identity = 10.

2.2.2 Enhanced Spatial Attention CycleGAN implementation

Figure 4 shows the ESA-CycleGAN model, which was developed as an extension of CycleGAN to address the weaknesses in unpaired image-to-image translation [28]. These problems include loss of edge details, fine textures, and pattern consistency in the style transfer results. But this model is highly relevant for converting flora images into Madura Batik motifs. That is because the conversion process requires the preservation of the natural structure of the flora objects (e.g., the contour of flowers or leaves), while also being able to incorporate the detailed ornamental patterns characteristic of Madura Batik. The ESA-CycleGAN architecture is based on the CycleGAN structure, which comprises two generators (G and F) and two discriminators ($D_X$ and $D_Y$). However, the ESA-CycleGAN architecture is enhanced with two additional modules: the edge extraction module and the self-attention module. The edge extraction module adds a Canny edge detection map to the input of the generator, assisting the network in identifying contours and significant features in the source image. It provides contour information that guides the generator in retaining the basic shape of the flora during the transformation process. While the self-attention module is incorporated into the generator and discriminator to model dependencies between spatially distant regions of the image. When combined, they make the ESA-CycleGAN able to learn global color and texture and, on top of that, maintain precise spatial detail, resulting in the generated batik motif often appearing more harmonious throughout the image.

Figure 4. Enhanced Spatial Attention CycleGAN (ESA-CycleGAN) flora and batik Madurese

This makes ESA-CycleGAN an effective architecture for producing richly ornamented Madura batik motifs while retaining the original flora’s natural character.

The Self-Attention Mechanism shows that the feature maps have three channels. The corresponding feature maps are produced by convolving the input with three 1 × 1 convolutional kernels for each branch. The feature map produced from the channel is transposed and calculated, subsequently undergoing matrix multiplication with the channel to yield a matrix resembling the covariance matrix, from which pixel correlation and similarity are evaluated. Thus, the attention feature map, particularly the feature weights, is obtained from the Softmax output. Ultimately, it is multiplied pixel by pixel with the channel’s feature map to obtain the attention-weighted feature map, computed as Eq. (4).

$O_j=\sum_{i=1}^N \frac{\exp \left(f\left(x_i\right)^T g\left(x_j\right)\right)}{\sum_{i=1}^N \exp \left(f\left(x_i\right)^T g\left(x_j\right)\right)} \times h\left(x_i\right)$      (4)

• $f\left(x_i\right)$: the result of a $1 \times 1$ convolution on the input features to generate the query vector at position $i$.
• $g\left(x_j\right)$: the result of a $1 \times 1$ convolution on the input features to generate the key vector at position $j$.
• $h\left(x_i\right)$: the result of a $1 \times 1$ convolution on the input features to generate the value vector at position $i$.
• $\exp \left(f\left(x_i\right)^T g\left(x_j\right)\right)$: computes the similarity between positions $i$ and $j$ using the dot product.
• The numerator and denominator form a Softmax function, ensuring that the attention weights at each position are positive and normalized.
• $O_j$: the final output of the attention aggregation process, representing a new feature representation at position $j$ after considering the contributions of all positions $i$ in the image.

ESA-CycleGAN expands the original CycleGAN model by adding two essential components to further reinforce structural preservation in its style translation: a canny-based edge extraction module and a self-attention module. The generator maintains the nine-residual-block ResNet-based encoder–residual–decoder structure, as summarized in Table 3, with the addition of self-attention after downsampling to capture long-range dependencies and preserve global motifs’ coherence. It also takes a concatenated edge map in addition to the RGB input to strengthen its retention of fine structural boundaries, such as the contours of the petals and ornamental edges. Meanwhile, the discriminator is based on a modified 70 × 70 PatchGAN architecture and has added a self-attention layer before the last convolutional block; see Table 4 for the improved sensitivity toward spatial relations and micro-texture realism typical of Madurese batik.

Table 3. Enhanced Spatial Attention CycleGAN (ESA-CycleGAN) generator architecture used for flora-to-batik style translation

Table 4. ESA-CycleGAN discriminator architecture used for flora-to-batik style translation

Training is done with least squares GAN loss to encourage stable adversarial learning; it combines a cycle-consistency loss (λ_cycle = 10), to ensure structural fidelity and an identity loss (λ_identity = 10), to stabilize color consistency. The model was trained for 50 epochs using the Adam optimizer (learning rate 2 × 10⁻⁴, β₁ = 0.5, β₂ = 0.999) with a batch size of 4.

2.2.3 Adaptive Instance Normalization CycleGAN implementation

AdaIN-CycleGAN is an extension of the CycleGAN architecture (Figure 5). This model enhances the generator by integrating the AdaIN mechanism directly into its residual blocks [29, 30]. Embedding AdaIN within the residual blocks allows the network to adaptively control style statistics (modifying the mean and variance of feature maps) while preserving the essential content structure.

Figure 5. Adaptive Instance Normalization CycleGAN (AdaIN-CycleGAN) flora and Madurese batik

This model relies on the AdaIN-CycleGAN architecture. That is because it works on two main ideas. The first is cycle consistency, which requires the network to recover the original image after a round-trip translation between domains. Second, AdaIN is used to align content features with style characteristics in a controlled manner. By combining cycle consistency and AdaIN, the model performs unpaired image-to-image translation that smoothly applies stylistic elements without losing the important structural details. For translating tasks with a large visual difference between source and target domains, this design is very suitable. For example, this approach can convert flora images into Madura Batik motifs.

AdaIN is analogous to style transfer. It adjusts the normalized channel-wise mean and variance of the content image to match those of the styled image, ensuring that the output image exhibits the same feature distribution as the ink painting. AdaIN lacks any trainable affine parameters. The supplied style image adaptively creates affine parameters. If x and y denote the feature maps of the content and style images, respectively, the AdaIN layer is as in Eq. (5):

$\operatorname{AdaIN}(x, y)=\sigma(y) \times\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\mu(y)$     (5)

• $x$: Feature from the content image (for example, a flora image).
• $y$: Feature from the style image (for example, a Madurese batik motif).
• $\mu(x)$: The mean value of the content feature $x$.
• $\sigma(x)$: The standard deviation of the content feature $x$.
• $\mu(y)$: The mean value of the style feature $y$.
• $\sigma(y)$: The standard deviation of the style feature $y$.

AdaIN-CycleGAN is an adaptation of the CycleGAN framework that integrates AdaIN to provide a more flexible and adaptive style control during floral-to-batik translation. Similar to the baseline architecture, the generator uses a ResNet-based encoder–residual–decoder structure with nine residual blocks, as summarized in Table 5. However, AdaIN-CycleGAN inserts AdaIN layers inside the residual block instead of the standard residual blocks, allowing the mean and variance of the feature maps to be dynamically aligned with the target batik style. This mechanism has facilitated smoother and more controlled style blending, especially in the case of floral textures combined with the color and ornamentation patterns of Madurese batik. As summarized in Table 2, the discriminator uses a similar PatchGAN-based architecture taken from CycleGAN, which evaluates local texture patches to ensure that the generated batik motifs preserve realistic micro-structural details.

Table 5. AdaIN-CycleGAN generator architecture used for flora-to-batik style translation

Training is set up similarly to CycleGAN with least squares GAN loss for stable optimization and cycle-consistency loss to preserve structural integrity during the bidirectional translation. Accordingly, AdaIN-CycleGAN was trained, according to the training script, for 50 epochs using the Adam optimizer (learning rate 2 × 10⁻⁴, β₁ = 0.5, β₂ = 0.999) with a batch size of 1, and only cycle-consistency loss without any identity loss was used. These settings ensure stable adversarial learning while still allowing AdaIN to more adaptively modulate style statistics during the floral-to-batik translation.