Enhanced Image Super Resolution Using ResNet Generative Adversarial Networks

The researchers introduced an image SR method utilizing directional bicubic interpolation [6]. Bicubic interpolation is a common technique in image interpolation due to its low complexity and fairly good results. However, it typically operates only in horizontal and vertical directions, which can lead to issues such as blocking, blurring, and ringing along edges. Various methods for interpolating lost pixels were used depending upon local strengths as well as directions. The approach maintains sharp edges and details better than bicubic interpolation. The approach outperforms previous edge-governed interpolations with regard to subjective, objective measurements as well as it has a low computational complexity.

A lightweight hybrid dense with residual connections was used, evaluating the balance between network complexity and performance enhancement [7]. When compared to earlier approaches, the suggested methods can greatly lower both the memory requirement and the inference speed to hold parameters as well as intermediate feature maps while keeping equivalent picture quality. Even though increasingly deep neural network models outperform conventional approaches in terms of SR, however to the large network parameters and convolution operations of deeper and denser networks, it is hard to execute them on low computational complexity, limited power, and low-memory systems.

The researchers introduced a novel algorithm that combines traditional methods with deep learning, leveraging the autonomous feature extraction capability of deep learning to embark on more profound reconstructions of low-resolution images [8]. The utilization of deep learning techniques, in conjunction with traditional interpolation methods, facilitated the training and learning processes, resulting in the production of high-resolution reconstructed data. This proposed method surpasses both standard interpolation algorithms and standalone deep learning approaches. Moreover, it excels in reconstructing intricate details, yielding sharper outlines and superior image quality. CDA's strong representational abilities and adaptable design ensure accurate matching of LR and HR image representations [9]. Its goal is to develop compact representation of input by keeping most crucial data.

The model forecasts the high frequency residuals in a fine way, and robust Charbonnier loss function providing close supervision [10]. The network architecture is generic; thus, it might be used to solve different image modification and synthesis issues. CNN designed with array of up-scaling filters has proved successful in displaying layer activations and creating semantic segmentations based on network high-level features [11].

It introduced RCAN with multiple residual groups interconnected by long skip connections for selective rescale of channel features while considering channel interdependencies [12].

Dense Nets inter layer connection sequence has been used for solving vanishing gradient problem, improve quality of feature, as well as promote feature reuse. Furthermore, it is suggested that DBPN be enhanced by incorporating dense connections in projection units, resulting in Dense DBPN. Dropout and Batch-Norm, that aren’t ideal for SR because they lose range flexibility over features, were avoided, unlike in the original Dense Nets. Instead, before entering the projection unit, a 1×1 convolution layer is applied for feature-pooling as well as dimensionality reduction. In D-DBPN, each part’s input is concatenation of all preceding units’ outputs. On huge scaling factors like 8X enlargement, the suggested network that surpasses state-of-art approaches such as SRCNN and DRRN [13].

Single scale and multi scale architectures were designed for reconstructing single SR and several scales of HR pictures in a model respectively [14]. Since Batch-Norm layers uses similar extent of memory as previous convolutional layers, GPU memory usage is likewise lowered. The proposed residual blocks are used to build a baseline (single scale) model. Although topology is similar to that of SRResNet, the model lacks ReLu activation external to the residual block. The multiscale representation adeptly manages various SR scales within a unified framework.

When interpolation methods are used to improve image resolution, they cause severe losses in their HF components. This may be seen in the smoothing that occurs as a result of the interpolation technique. The suggested image enhancement method aims to deliver higher resolution than existing SR-DWT and DASR. It forms necessary to preserve the borders and fine characteristics of an image in order to improve its resolution. The main difference among designed method and other standard algorithms is better edge and fine feature preservation, i.e., producing a sharp optimized image through intermediary stage in the HF sub-band interpolation procedure, and able to perform the disparity among the LL and the LR pictures [15].

To generate a high-quality photo, the researchers focused on developing and refining perceptual loss functions derived from extensive features of pre-trained networks [16]. By utilizing these loss functions, they trained forward networks for various image enhancement tasks, effectively merging the strengths of both methods. Their approach showcased results in photo style transfer, where a feed-forward network was trained to address enhancement challenges in real-time.

The researchers introduced SinGAN, a generative model capable of being trained unconditionally on a single natural image [17]. This model is designed to learn the internal distribution of patches within an image, allowing it to generate high-quality, diverse samples that maintain similar visual content. SinGAN consists of a pyramid of fully convolutional GANs, each responsible for capturing the patch distribution at different scales of the image. The authors employed a compact hourglass-shaped CNN architecture to accelerate and improve super-resolution (SR), enhancing the performance of the existing SRCNN model [18].

The SRCNN architecture was redesigned in three key aspects. First, a deconvolution layer was added at the end of the network to allow for direct learning of mappings from the low-resolution image to the high-resolution image. Second, the mapping layer was restructured by initially reducing the input feature dimension and then expanding it afterward. Finally, they adopted smaller filter sizes and increased the number of mapping layers in the third step.

The researchers introduced PULSE, which is a latent oriented up-sampling super-resolution method that generates high-resolution, realistic images by exploring the HR natural image manifold [19]. This self-supervised approach is not restricted to a single degrading operator used during training and focuses on creating images that downscale accurately to the original low-resolution input.

Even before training, the structure of the generator network could capture numerous low-level image details. In inverse tasks such as denoising, super-resolution, and inpainting, a randomly initialized neural network effectively serves as a prior, achieving remarkable results. The identical prior can even be utilized to analyze deep neural representations also restore images from flash-no-flash input pairs using the identical prior. Inductive bias is encapsulated within traditional generator network architectures [20].

The researchers proposed a self-supervised method with temporal self-containment, crucial for achieving temporally clear solutions without sacrificing spatial textures [21]. They introduced the Ping-Pong loss to enhance temporal consistency over time, preventing recurrent networks from accumulating artifacts while preserving detailed features. Additionally, they introduced a comprehensive set of metrics to objectively evaluate the correctness and perceptual quality of temporal evolution.

A novel architecture aims to maintain spatially accurate high-resolution representations while extracting significant contextual information from low-resolution representations. This approach features a multi-scale residual block with concurrent multiresolution convolution flows for multi-scale feature retrieval, cross-flow information exchange, spatial and channel attention mechanisms for context acquisition, and attention-based multi-scale feature aggregation. In essence, it learns an extensive set of features across various scales while preserving high-resolution spatial details [22].

The researchers demonstrated that models with wider features before ReLU activation significantly enhanced SISR performance without increasing parameters or computational costs [23]. A novel projection-based method to incorporate conditional information into the GAN discriminator without altering its functionality in the probabilistic model was proposed [24]. The researchers introduced EDVR, a Video Restoration framework using Enhanced Deformable Convolutions [25]. They developed the PCD alignment module, which handles large motions through pyramid, cascading, and coarse-to-fine deformable convolutions for precise feature alignment.

ResNet-GAN is a technique used for super-resolution image generation, transforming low-resolution (LR) images into high-resolution (HR) ones. It leverages ResNet architecture for learning residual functions and generative adversarial networks (GANs) for generating realistic HR images. This approach enhances the quality and resolution of images, making them suitable for various applications requiring high-quality visuals.

4.1 Generative adversarial network with residual learning

A GAN comprises a Generator and a Discriminator. The Generator aims to craft realistic fake images, while the Discriminator discerns these fakes. This adversarial strategy is well-suited for generating high-quality SR images, as the iterative interaction between the Generator and Discriminator enhances results. GANs offer a robust framework for generating natural-looking, high-quality images. The GAN process guides reconstructions towards regions likely to contain photorealistic images, closely aligning them with the original image manifold. This paper introduces the first deep-ResNet framework that employs GANs to implement a perceptual loss function for photorealistic single image SISR. The Res-Net GAN, tailored for this purpose, integrates a novel perceptual loss by replacing the traditional MSE-based content loss with one computed using VGG network feature maps, which are more attuned to changes in pixel space.

4.1.1 Generator

The Generator part in the GAN architecture here consists of Residual Network (ResNet) model for image processing. ResNet, a prominent neural network architecture in deep learning, is widely employed across various image processing tasks. As subsequent winning architectures incorporate more layers to reduce error rates in deep neural networks, they encounter a common challenge known as the vanishing/exploding gradient problem. This issue arises when the gradient becomes either excessively small or large, leading to increased training and test error rates with an increasing number of layers. To address the vanishing/exploding gradient problem, ResNet introduces the concept of residual learning, which involves utilizing skip connections in the network architecture. These skip connections bypass certain layers during training and connect directly to the output.

Residual learning, implemented in ResNet, involves learning residual functions instead of direct mappings. Within residual learning, there are two types: global residual learning and local residual learning. In the suggested model, local residual learning is applied, which aids training by reducing task complexity and enhancing learning rates in deep learning. Shortcut connections between various layers within the network, of varying depths, are utilized in conjunction with element-wise addition to facilitate local residual learning.

4.1.2 Discriminator

The discriminator in a GAN represents a classifier which distinguishes between original data and data fed by the generator. The selection of the network architecture can be based upon the type of data. The generator generates fake data instances, and these are used by the discriminator as negative examples during training. Sigmoid function to classify the image whether they are similar or not.

1.png

Figure 1. Discriminator architecture [29]

Because the purpose of discriminative models is to detect fraudulent data, the discriminative neural network is trained to reduce the final classification error. It learns to distinguish between the various classes by examining both genuine and fake samples generated by the generator and attempting to determine which are real and which are fake. Figure 1 shows how the discriminator of GAN architecture differentiates between HR and SR images.

The output of the discriminator, a binary value indicating real or fake, guides the generator in GANs to produce high-resolution images from low-resolution inputs. When the discriminator identifies a generated image as realistic, the generator refines its parameters to produce even higher-quality images. Conversely, if the discriminator detects flaws, the generator adjusts to enhance image quality. This iterative process enables the generation of more realistic high-resolution images from low-resolution inputs.

4.2 ResNet GAN model

Res-Net GAN is a state-of-the-art method for producing photo-realistic super-resolution(SR) images with high upscaling factors(x4), validated by exhaustive MOS analysis. The goal of SISR is to generate a high-resolution(HR) image from a low-resolution(LR) input. During training, HR images are available, and LR equivalents are created by applying a Gaussian filter and downsampling.

The primary objective is to train a generator function (G) that estimates the HR counterpart of a given LR image using a feedforward CNN. The generator network consists of multiple identical residual blocks, each with convolutional layers, batch normalization, and parametric ReLu. The input LR image is processed through these residual blocks, followed by pixel shufflers to increase the resolution, and a final convolutional layer.

The discriminator network is trained to distinguish real HR photos from generated SR images. It uses leaky ReLu activation and strided convolutions to reduce picture resolution, similar to the VGG network, and includes dense layers for classification.

The performance of the generator network relies on a perceptual loss function, combining Mean Square Error (MSE) and VGG feature maps. This approach encourages the generator to produce images that align closely with the original image manifold, aiming to deceive the discriminator model.

The architecture aims to generate high-quality SR images by promoting perceptually superior solutions found within the natural image subspace, as shown in Figure 2.

2.png

Figure 2. Schematic diagram of Res-Net GAN [29]

First, the dataset requiring higher resolution images is collected. Use the generator function to process them. This generator function boosts the image’s resolution. The image is then sent to the discriminator function. The discriminator determines whether the image is genuine or not and returns a binary result. Return the picture to the generator function if the binary value is 0. Generator boosts the resolution once more and sends it to the discriminator, it checks the image once more. This procedure is repeated until a high-resolution image is obtained. Return the image if the discriminator’s output is 1.

4.2.1 Training details

The NVIDIA Tesla M40 GPU was utilized to train the networks using a random sample SR image set from Div2k [30]. These images differ from those used in testing. Low-resolution (LR) images were created by downsampling high-resolution (HR) images (BGR, C = 3) with a bicubic kernel and a downsampling ratio of r = 4. Each mini-batch involved randomly cropping sixteen 96×96 HR sub-images from various training photos. As the generator model is entirely convolutional, it can be applied to images of any size. LR input images were scaled to [0, 1], while HR images were scaled to [1, 1]. MSE loss was computed on images with an intensity range of [1, 1]. The rescaling of VGG feature maps by a factor of 1 to 12.75 aims to align the scale of VGG losses with the scale of Mean Squared Error (MSE) loss. This rescaling factor ensures that the magnitudes of the losses are comparable, allowing for more balanced optimization during training. By using a rescaling factor of 0.006, this alignment is achieved, ensuring that both types of losses contribute effectively to the overall optimization process without one dominating over the other. Utilizing Adam with β = 0.9 for optimization, provides a balance between adaptability and momentum making it effective for optimizing the training of deep neural networks when used in super-resolution tasks. SRResNet networks were trained with a learning rate of 10⁴ and 10⁶ update iterations. To avoid undesirable local optima, when training the GAN, MSE-based SRResNet was used as initialization for the generator. Various SRGAN variants were trained using 10⁵ update iterations and learning rates of 10⁴ and 10⁵. Generator and discriminator networks were alternately updated with k = 1. Remaining blocks in the generator network were identical (B = 16). Batch normalization updates were disabled during testing to ensure deterministic results dependent only on the input.

4.3 Loss functions

It’s a way of determining how well your algorithm models the data. At its most basic level, a loss function is a measurement of how well your prediction model predicts the expected outcome (or value). The learning problem is turned into an optimization problem, a loss function is established, and the algorithm is tuned to minimize the loss function. The difference between the expected output and the actual output of the machine learning model is computed which facilitates the gradient computation. Subsequently, the weights can be updated from the loss function. The cost is calculated as the average of all losses. There are three types of loss functions, they are: Perceptual loss, Content loss and Adversarial loss.

4.3.1 Perceptual loss

Below, the perceptual loss function utilized is elaborated upon. The performance of our generator network relies on super-resolution (SR). While Mean Squared Error (MSE) is often utilized for low-resolution to high-resolution (l-SR) tasks, it lacks the ability to evaluate solutions based on perceptually significant qualities. To address this, we introduce a loss function that considers perceptual aspects. This comprises a weighted sum of content loss (l-SR-X) and an adversarial loss component to compute the perceptual loss.

In particular, our focus lies on image transformation problems, where an output image is generated from a modified input image. Recent methodologies tackling such challenges typically involve training feedforward convolutional neural networks using a loss measured in pixels per pixel, comparing the output to the ground-truth original images. We propose the utilization of perceptual loss functions for training feedforward networks designed for image transformation tasks. This approach amalgamates the benefits of both pixel-wise comparison and perceptual evaluation, potentially enhancing efficiency. Our feedforward network is trained to address optimization problems in real-time, demonstrated notably in the context of visual style transfer within this paper. Comparing our approach with optimization-based methods reveals qualitatively equivalent outcomes, achieved at a speed three orders of magnitude faster than the latter. As an additional experiment, we explore single-image super-resolution, substituting perceptual loss for per-pixel loss, resulting in visually pleasing results.

4.3.2 Content loss

It is the prevailing optimization target in image super-resolution (SR), widely employed by state-of-the-art approaches. Nonetheless, solutions stemming from mean squared error (MSE) optimization often yield high peak signal-to-noise ratio (PSNR) but lack high-frequency content, leading to visually displeasing outcomes characterized by overly smooth textures. Instead of relying solely on pixel-wise losses, we opt for a loss function that prioritizes perceptual similarity, thereby mitigating the issue of excessively smooth textures and to measure how well the high-resolution output image preserves the structural content of the low-resolution input image.

4.3.3 Adversarial loss

A continuously trained discriminator network defines the adversarial loss. It’s a binary classifier that distinguishes between real-world data and data generated by a generative network. We add the generative block of our GAN algorithm to the perceptual loss function in addition to the losses from content discussed so far. By attempting to trick the discriminator network, this encourages the network to favor solutions which are based on a variety of natural images. The loss from generative function l-SR-Gen is calculated using the discriminator DD (GG (I-LR)) probabilities across all training samples.

4.4 ResNetGAN for SR algorithm

Take image data set to enhance resolution.

Pass through Generator function.

def Gen():

// code to increase resolution of the input image.

//Return this to discriminator function.

Take input to discriminator function to check whether the image is real or fake.

def Dis():

// code to check real or fake.

// if it is fake return that to generator to improve quality.

// if it is real, return to the output as SR images.

First take the data set for which you want to get the super resolution images. Pass them through generator function. This generator function increases the resolution of the image. Then it sends that image to discriminator function. Discriminator checks whether the image is real or fake and generates a binary value. If the binary value is 0 return that image to generator function. Generator again increases the resolution and sends it to discriminator and again discriminator checks the image. This process continues until you get a high-resolution image. If the output of discriminator is 1 then return the image.

The Res-Net Generative Adversarial Network (GAN) has demonstrated highly effective in enhancing low-resolution images, establishing its superiority over traditional ResNet through comprehensive comparisons across four different parameters based on the perceptual quality of super-resolved images. The study focuses on the perceptual quality of super-resolved images over computational efficiency. The proposed approach plays a vital role across diverse applications in computer vision field. It can enhance the resolution of medical scans for precise diagnosis, improve satellite imagery for environmental monitoring, aid surveillance for better object identification, and enhance digital media content for superior visual experiences.

Furthermore, our study detailed SRResNet, achieving state-of-the-art performance in image super-resolution based on PSNR evaluation. We introduced SRGAN, combining content loss and adversarial loss for improved realism, particularly evident in large upscaling factors through MOS testing. We emphasized the limitations of PSNR and SSIM metrics in capturing perceptual image quality. While shorter networks offer efficient alternatives with minimal degradation, deeper designs benefit performance, influenced by ResNet architecture. Deeper networks enhance SRResNet performance, though longer training times are required. However, deeper SRGAN variations face challenges due to high-frequency aberrations in data. The optimal loss function varies based on application needs. Achieving perceptually convincing image reconstructions is a significant challenge for future research, involving the development of algorithms for real-time processing in fields like video streaming and surgery, handling dynamic environments in surveillance and autonomous vehicles, addressing limited annotated data in specialized fields like medical imaging, mitigating noise and artifacts in satellite imaging and microscopy, incorporating semantic understanding for scene understanding and medical diagnostics, and ensuring adaptability to diverse domains such as urban environments and industrial settings. Designing content loss functions that take into account the spatial content of the image while maintaining uniformity to changes in the pixel space will likely enhance the photorealistic results of images even further in the future by employing variants of CNNs, along with regularization techniques.