A Positive-Unlabeled Generative Adversarial Network for Super-Resolution Image Reconstruction Using a Charbonnier Loss

High-resolution images, which are rich in texture details, have gained popularity in many fields, such as remote sensing, medical imaging, safety monitoring, and various visual scenes (e.g., industrial scene). The super-resolution reconstruction of high-resolution images from low-resolution images is a typical ill-posed problem.

Recent studies have proved the superiority of convolutional neural network (CNN) in super-resolution reconstruction of high-resolution images [1-3]. In many CNN-based super-resolution methods, the pixel loss is adopted to find the quantitative training indices that improve the peak signal-to-noise ratio (PSNR). Although it can be optimized easily, the pixel loss rarely provides gratifying real details in the light of perceptual vision, which distorts the reconstructed image. The distortion is particularly severe for big scaling factors.

The generative adversarial network (GAN) [4, 5] performs well in generating realistic images, providing a new method for perceptual super-resolution imaging. Ledig et al. [6] attempted to minimize perceptual loss and adversarial loss, and used 16 depth residual blocks and sub-pixel convolution layers to deal with one big scaling factor (×4). Their attempts significantly improved the rich visual perception. However, the GAN-based super-resolution methods have such defects as vanishing gradients, mode collapse, and optimization difficulty in GAN training. As a result, these methods are more likely to collapse quickly, and suffer from over-fitting than CNN-based super-resolution methods [7, 8].

To overcome the above defects, Öcal and Özbakır [9] introduced supervised learning into neural networks. But supervised learning requires a huge amount of training data. To reduce the dependence on numerous training data, Zuo et al. [10] implemented the automatic encoder (AE) in neural network design to learn data feature automatically, and obtain the representation space of low-dimensional potential data. In addition, GAN was combined with AE into a new hybrid generation model to reconstruct high-dimensional data. Nevertheless, there is a contradiction between the fidelity and variety of the potential data representation.

To balance fidelity with variety, Brock et al. [11] applied orthogonal regularization to the generator, making it applicable to simple truncation. Then, the trade-off between sample fidelity and variety could be finetuned by truncating the latent space. Yet this strategy needs a careful design of the network architecture. To avoid the problem, Arjovsky et al. [12] and Gulrajani et al. [13] tried to stabilize the generation process by fitting and optimizing the Wasserstein distance. Miyato et al. [14] proved the necessity and benefits of introducing Lipschitz continuity into the discriminator.

Overall, the existing single image super-resolution methods are unable to deal with multiple degradation well, and poor in generalization. The generated images are often blurry and over smooth. To realize the super-resolution reconstruction of multiple degradation, Lin et al. [15] fully utilized the prior knowledge of fuzzy kernel and noise level in the generation network, employed three discriminators of different scales to reconstruct image details, and thereby maintained the global consistency of the reconstructed image.

The loss functions of the above methods lack texture-oriented optimization, failing to completely maintain the structural information of geometric texture. To improve the perceptual quality of reconstructed images, some researchers regarded regularization as the ultimate solution, the most popular of which is total variation [16-18]. In the context of images, total variation is based on the following fact: an image with excessive and possible false contents, such as noise and unusual artifacts, has a high total variation. This brings an extremely high absolute gradient of the image. The total variation must be minimized to restore the original high-resolution image. In this way, the key features of an image, namely, edges and contours, can be retained, while removing the unnecessary details. Albeit the above strengths, total variation regularization has the following disadvantages: Firstly, false edges often appear in highly noisy images. Secondly, the edges with obvious curvature may be oversmoothed, and the small-scale features may be destroyed. Thirdly, the contrast ratio and geometry structure of the final output may get lost [19].

Perona and Malik [20] proposed another regularization method based on diffusion process: the diffusion rate in the image region is controlled spatially, using a diffusion coefficient. Hence, the outliers are removed without losing the important details of the image. The main shortcomings of the method are as follows: Their flux function is nonmonotonic and not completely convex. Theoretically, uncontrolled non-monotonic and non-convex functions are unstable, which may lead to infinite solutions about the initial image [21, 22]. Charbonnier et al. [23] proposed an alternative flux function that is always monotonic and convex. In practice, their function achieves better results than the Pernoma-Malik method with speckle effect. Zhang et al. [24] developed an enhanced Laplace pyramid network, and trained the network with multilevel CGAN, VGG and robust Charbonnier loss function, producing high-quality super resolution results. Ketsoi et al. [25] improved the feature pyramid network into the enhanced feature block network (EFBN), which adopts high-dimensional feature maps, and uses shared parameters among low-level feature maps. In this way, the computational complexity is reduced, and the perceived quality is improved for the reconstructed images.

At present, the discriminators of SRGAN methods regard the real data as positive samples, and the generated data as negative samples. The discrimination standard remains fixed throughout the training, without considering the gradual quality improvement of the generated data. In fact, the generated data may surpass the real data in terms of quality, making the training unstable. The trained discriminator should neither be too good nor be too bad. Excessively well training will lead to vanishing gradients, and excessively poor training will make it difficult to distinguish between good and bad generated samples. Ideally, the discriminator should be trained to the correct state, which is no easy task in training. What is worse, common measures like PSNR and SSIM are not suitable for measuring the perceptual similarity acceptable to human vision.

To stabilize the discriminator training and ensure the visually perceptibility of reconstructed images, this paper utilizes the positive unlabeled GAN for single image super-resolution reconstruction using a Charbonnier loss function. The main contributions of this paper are as follows:

(1) Taking the Charbonnier penalty function as the loss function, the discriminator was designed to improve the training stability.

(2) The generated super-resolution images were processed as unlabeled samples, making the generator focus on improving the low-quality generated super-resolution images. This improves the performance of the generator.

(3) The real texture, background, and outline details of the reconstructed image were enhanced as much as possible by a new set of perceptual loss, i.e., the weighted sum of content loss, feature loss, texture loss and Charbonnier relative adversarial loss.

This section details the proposed approach, namely, SRPUGAN-Charbon.

3.1 Objective loss function

The definition of perceptual loss function [31] is crucial to the performance of the generated network. Inspired by SRGAN [32], our object loss function includes content loss and adversarial loss:

$\ell=\ell_{\text {charbon }}+0.008 \ell_{\mathrm{VGG}-\text { charbon }}+2 \times 10^{-6} \ell_{\alpha \text {-charbon }}+10^{-3} \ell_{\mathrm{PU}-\mathrm{Gen}}$  ,         (3)

where, $\ell_{\text {charbon }}$  is the Charbonnier loss; $\ell_{V G G-c h a r b o n}$    is the improved VGG loss [33]; $\ell_{\alpha-\text { charbon }}$   is the a-Charbon loss; $\ell_{P U-G e n}$   is the adversarial loss.

3.1.1 Content loss

Mean squared error (MSE) is widely used to improve the PSNR of single image super-resolution reconstruction, but it cannot effectively handle outliers. Hence, a robust loss function $\ell_{\text {charbon }}$ was adopted to handle outliers:

$\ell_{\text {charbon }}=\frac{1}{\mathrm{r}^{2} \mathrm{WH}} \sum_{\mathrm{w}=1}^{\mathrm{rW}} \sum_{\mathrm{h}=1}^{\mathrm{rH}} \rho\left(\mathrm{y}_{\mathrm{w}, \mathrm{h}}-\mathrm{G}(\mathrm{x})_{\mathrm{w}, \mathrm{h}}\right)$,         (4)

where, r is the upsampling factor; W and H are the width and height of the high-resolution image, respectively; $\rho(m)=\sqrt{m^{2}+\varepsilon^{2}}, \varepsilon=10^{-3}$ is the Charbonnier penalty function [34]; x is the low-resolution image; y is the original high-resolution image.

Ledig et al. [32] successfully introduced VGG loss [33] into super-resolution reconstruction. This loss has been used not only for pixel level loss, but also for perceptual loss. The loss $\ell_{V G G-c h a r b o n}$ can be defined as:

$\ell_{\mathrm{VGG}-\text { charbon }}=\frac{\sum_{\mathrm{w}=1}^{\mathrm{W}_{5,4 \quad}} \sum_{\mathrm{h}=1}^{\mathrm{H}_{5,4 \quad}} \rho\left(\phi_{5,4 \quad}(\mathrm{y})_{\mathrm{w}, \mathrm{h}}-\phi_{5,4}(\mathrm{G}(\mathrm{x}))_{\mathrm{w}, \mathrm{h}}\right)}{\mathrm{W}_{5,4\quad} \mathrm{H}_{5,4\quad}}$        (5)

where, $\phi_{5,4}$ is the feature map obtained after the rectified linear unit (ReLU) of the 4^th convolutional layer [35] and before the 5^th max pooling layer in VGG19 [33]; $W_{5,4}$ and $H_{5,4}$ are the width and height of the corresponding feature map, respectively.

Further, a-Charbon regularization was adopted to preserve the good, clear and distinguishable local structure (e.g., edges) of the generated image. The regularization term $\ell_{\alpha-\text { charbon }}$ [30] can be defined as:

$\ell_{\alpha \text {-charbon }}=\frac{1}{\mathrm{r}^{2} \mathrm{WH}} \sum_{\mathrm{w}=1}^{\mathrm{rW}} \sum_{\mathrm{h}=1}^{\mathrm{rH}} \frac{\nabla_{\theta_{\mathrm{G}}} \mathrm{G}(\mathrm{x})_{\mathrm{w}, \mathrm{h}}}{\left(1+\left[\frac{\nabla_{\theta_{\mathrm{G}}} \mathrm{G}(\mathrm{x})_{\mathrm{w}, \mathrm{h}}}{\mathrm{k}}\right]^{\alpha}\right)^{\frac{1}{\alpha+\varepsilon}}}$,         (6)

where, θ_G is the parameter of the generator; 0≤α≤2.

3.1.2 Adversarial loss

Referring to SRGAN [32], the authors added the generation loss to the perceptual loss. The aim is to encourage the network to support the solution residing on the natural image manifold by cheating the discriminator. The specific process is as follows: add one unknown class priori [26] penalty term to the generation loss, treat the generated samples as unlabeled, and make the generator focus on improving the low-quality generated samples. In this way, the generated network is optimized effectively to realize better performance.

The adversarial loss can be defined as:

$\ell_{\mathrm{PU}-\mathrm{Gen}}=\frac{1}{\mathrm{n}} \sum_{\mathrm{i}=1}^{\mathrm{n}}\left(\log \left(1-\mathrm{D}\left(\phi\left(\mathrm{y}_{\mathrm{i}}\right), \phi\left(\mathrm{G}\left(\mathrm{x}_{\mathrm{i}}\right)\right)\right)+\varepsilon\right)+\lambda\left[\| \nabla_{\theta_{\mathrm{d}}}\left(\pi \log \left(\mathrm{D}\left(\phi\left(\mathrm{y}_{\mathrm{i}}\right)\right)\right)+\max \left(\begin{array}{c}0, \log \left(\mathrm{D}\left(\phi\left(\mathrm{G}\left(\mathrm{x}_{\mathrm{i}}\right)\right)\right)\right) \\ \left.\left.-\pi \log \left(\mathrm{D}\left(\phi\left(\mathrm{G}\left(\mathrm{x}_{\mathrm{i}}\right)\right)\right)\right)\right)\right) \end{array}\right)\right)\|_{2}-1\right]^{2}\right)$            (7)

where, $\varepsilon=10^{-2}$ is a parameter to prevent the logarithmic term from being zero; θ_d is the parameter of the discriminator; π is the a priori knowledge of the class, i.e., the proportion of positive data (high-quality data) in unlabeled data; n is the number of training samples; λ is a regularization parameter.

3.2 Network structure

Drawing on the above objective losses, the network structure was determined for our approach (Figure 1).

The proposed approach SRPUGAN-Charbon can be trained in five steps:

Step 1. A low-resolution image x is imported to the generator network, and the good, clear and distinguishable local structure of the generated image is saved by a-Charbon regularization to obtain the corresponding reconstructed image G(x). Then, the content loss between the real high-resolution image y and the reconstructed image G(x) is calculated by Charbon penalty function.

Step 2. The real high-resolution image y and the reconstructed image G(x) are imported into VGG to extract their high-level features, ϕ(y) and ϕ(G(x)). Similarly, the content loss between high-level features ϕ(y) and ϕ(G(x)) is calculated by the Charbon penalty function.

Step 3. The high-level features ϕ(y) and ϕ(G(x)) are inputted into the discriminator network, and the adversarial loss is obtained through PU classification regularization. The final objective loss function is the weighted sum of content loss and adversarial loss.

Step 4. Backpropagation of the network is realized by the adaptive a-Charbon method and PU classification regularization. During the error backpropagation, the gradient of each layer is calculated, and the network is optimized by iteratively updating the discriminator network parameter θ_d and the generator network parameter θ_Gaccording to the training strategy.

Step 5. The above steps are repeated until the loss function reaches its minimum, marking the end of network training.

1.png

Figure 1. Network structure

This paper proposes a PUGAN-Charbon-based single image super-resolution reconstruction method. Our method includes a generator network for producing super-resolution images, and a discriminator network for judging quality of the generated super-resolution images. The PUGAN model is adopted in the discriminator network to encourage the generator to produce better images, by judging whether the quality of the generated image is good or bad, rather than whether the generated image is from super-resolution image or real high-resolution image. Compared with other GAN models, the PUGAN model enhances the stability of GAN training. To reconstruct the low-frequency and high-frequency features of the low-resolution image, the authors synthetized content loss, adversarial loss and Charbonnier loss into the objective function. Content loss and adversarial loss are conducive to reconstructing the high-frequency features of the low-resolution image, such as textures and edge details, while Charbonnier loss helps to reconstruct the low-frequency features of the low-resolution image, such as image contour and background, by removing outliers. Extensive experiments on three benchmark databases show the advantages of the proposed SRPUGAN-Charbon method over other state-of-the-arts, as indicated by the perceived quality, PSNR and SSIM of reconstructed images.