A Multi-Scale Image Enhancement Algorithm Based on Deep Learning and Illumination Compensation

Image is the main carrier and primary way of expression for visual information [1-9]. When the illumination is insufficient or ununiform, or blocked by objects, the captured images often have visual problems like as low brightness, high noise, and low contrast [10-17]. High-quality images are always expected, whether in human visual perception or in image applications [18-25]. If a dark low-illumination image, which does not conform to the visual perception of the human eyes, can be easily converted into a normal light image, it would bring great convenience to our daily life.

Low or uneven brightness reduces the contrast of near-infrared and optical remote sensing images, making it challenging to analyze the image contents. Huang et al. [26] proposed a spatially adaptive method to enhance multi-scale images: the nonsubsampled contourlet transform is adopted to decompose each low-contrast image into multiple scales; based on improved histogram equalization, a spatially adaptive gamma correction strategy is presented to enhance the base layer, which serves as the guide layer. Zhou et al. [27] applied the style transfer of CycleGAN (generative adversarial network) to low-illumination image enhancement. During the network structural design, different convolution kernels were used to extract features from three paths, and a deep residual shrinkage network was developed to suppress the noise after convolution. Both visual effects and objective indices show that the CycleGAN, which is based on multi-scale deep residual shrinkage, performs excellently in low-illumination enhancement, detail restoration, and denoising. Zhou et al. [28] proposed a multi-scale Retinex-based adaptive grayscale transform approach for underwater image enhancement. Their approach consists of three steps: color correction, image denoising, and detail enhancement. For different degraded underwater images, simulated annealing optimization was introduced to implement adaptive grayscale transform, and enhance image details. Pan et al. [29] created a multi-scale fusion residual encoder-decoder approach for low-illumination image enhancement. The approach directly learns the end-to-end mapping between the original light and dark images from the original sensor, and fully restores the details and colors of the original image, while effectively enhancing the image brightness. To improve the quality of image textures, Yuan et al. [30] put forward a multi-scale fusion enhancement method. Two new fusion inputs were obtained by different color methods. One input was utilized in the red-green-blue (RGB) model to improve image resolution before de-stacking, through the contrast-based dark channel. The other input was adopted for color compensation, based on multiple morphological operations and the opponent colors in the CIE 1976 L*a*b* color model.

The existing methods need to be improved in terms of detail processing and noise reduction. None of them fully consider the effects of nonuniform light intensity. Besides, these methods ignore the influence of incident light on real color images. As a result, the output images are often not effectively enhanced or not natural, and the enhancement effect is not ideal for multi-scale images. To solve the problem, this paper presents a multi-scale image enhancement algorithm based on deep learning and illumination compensation. Section 2 develops a compensation algorithm for image illumination drift based on the Hodge decomposition model. Section 3 enhances low-illumination multi-scale images by a lightweight multi-scale Retinex network, details the architecture of that network, and designs a new joint network loss function for the input and output forms, corresponding to the proposed deep learning network, and the low-illumination multi-scale image enhancement. Experimental results demonstrate the effectiveness of the proposed model.

If a low-illumination image enhancement algorithm boasts a good enhancement effect and a low time complexity, then the corresponding visual system will perform excellently. Both accuracy and real-timeliness are two key parameters of a good image enhancement algorithm. Otherwise, the algorithm would be unsuitable for wide application. Traditionally, low-illumination image enhancement methods are mostly based on physical models. Despite their good robustness and generalization ability, these methods cannot achieve ideal enhancement effect, when the external illumination is very complex. The deep learning-based image enhancement algorithms can realize the enhancement requirements of the visual system. In multi-scene and low-illumination conditions, the traditional deep learning models have a limited generalization ability, which does not apply to low-illumination images in a complex background. Besides, their algorithms consume too much time, failing to meet the real-time visual needs. Therefore, this paper chooses the lightweight multi-scale Retinex network to enhance low-illumination multi-scale images. The selected network can fully satisfy the actual needs, for its good performance and simple structure.

Figure 2 shows the architecture of lightweight multi-scale Retinex network, which encompasses four modules. To mine the image features of multiple scales, the Gaussian pyramid module performs multi-scale decomposition of the low-illumination images. To obtain the initial illumination, the illumination extraction module estimates and extracts the illumination information in multi-scale images. To obtain fused illumination, the illumination fusion module merges the extracted deep features and cross-scale features of multi-scale lights. To get fine illumination, the recovery module integrates and adjusts the fusion of multi-scale lights. Finally, the enhanced multi-scale image is obtained by Retinex transform. Figure 3 presents the multi-layer structure of the illumination extraction module.

2.png

Figure 2. Architecture of lightweight multi-scale Retinex network

3.png

Figure 3. Structure of the illumination extraction module

The single-scale Retinex algorithm is difficult to determine the scale parameter τ. This limitation can be overcome by the MSR-Net, a deep learning network based on the multi-scale Retinex algorithm. The key processing steps of the network can be expressed as:

$QU\left( a,b \right)=S\left( a,b \right)\times K\left( a,b \right)$     (12)

$s(a, b)=\log (S(a, b))=\log (Q U(a, b))-\log (K(a, b))$     (13)

$s\left( a,b \right)=log\left( QU\left( a,b \right) \right)-log\left( G\left( a,b \right)\otimes QU\left( a,b \right) \right)$     (14)

$s\left( a,b \right)=log\left( QU\left( a,b \right) \right)-{}^{1}/{}_{3}log\left( \sum\limits_{m=1}^{3}{{{L}_{m}}{{o}^{-\frac{{{a}^{2}}+{{b}^{2}}}{2\tau _{m}^{2}}}}}\otimes QU\left( a,b \right) \right)$     (15)

In deep learning models, the loss function guides the network training, exerting a huge impact on the mapping ability of the original input to the actual results. This paper designs a new joint network loss function for the input and output forms, corresponding to the proposed deep learning network, and the low-illumination multi-scale image enhancement. The function involves seven loss terms: trend consistency loss K_TC, color consistency loss K_CC, local loss K_LO, exposure control loss K_EC, mean squared error loss function K_JF, structural similarity loss function K_JG, and smoothing loss function K_PH. The weights of the corresponding loss terms in the joint loss function are represented by q_s, q_x, q_n, q_rr, and q_r, respectively. The joint network loss function can be given by:

$K={{K}_{TC}}+{{K}_{LO}}+{{q}_{s}}{{K}_{EC}}+{{q}_{x}}{{K}_{CC}}+{{q}_{n}}{{K}_{JF}}+{{q}_{rr}}{{K}_{JG}}+{{q}_{r}}{{K}_{PH}}$     (16)

Each loss term is detailed below. Firstly, the trend graph was calculated based on the traditional calculation of image gradient graph. Let ∇_cg(a,b) be the first-order gradient a certain direction c of the image g(a,b); f and q be the vertical and horizontal directions of direction c, respectively; p be a random integer. Then, the trend value of g(a,b) in a certain direction can be calculated by:

${{\bar{\nabla }}_{c}}g\left( a,b \right)=\left\{ \begin{align}  & p\text{  }\nabla {{g}_{c}}\left( a,b \right)>0 \\ & 0\text{  }\nabla {{g}_{c}}\left( a,b \right)=0 \\ & -p\text{  }\nabla {{g}_{c}}\left( a,b \right)<0 \\\end{align} \right\},c\in \left\{ f,q \right\}$     (17)

Let ∇^*Ṙ be the trend graph of the enhanced image Ṙ; ∇^*R be the mean of the trend graph of the original image R; ∇^*Ṙ_ij be the mean trend value of the neighborhoods of the same size in any area in Ṙ; ∇^*R_ij be the mean trend value of the local area corresponding to the same area in Ṙ. The value of μ₁ is determined through multiple tests. The trend consistency loss K_TC can be calculated by:

${{K}_{TC}}=\frac{1}{N}\sum\limits_{i=1}^{N}{\sum\limits_{j=1}^{9}{\left| {{\nabla }^{*}}{{{\dot{R}}}_{ij}}-{{\nabla }^{*}}{{R}_{ij}} \right|}}+{{\mu }_{1}}\times \left| {{\nabla }^{*}}\dot{R}-{{\nabla }^{*}}R \right|$    (18) 

The local computations are carried out in any number of areas to overcome the lack of representativeness of global operations in local areas. Let L be the number of randomly selected local areas; M(i) be the set of the four neighborhoods of the i-th local area; ∇^*Ṙ_i and ∇^*R_i be the mean gradients in the same local area of Ṙ and R, respectively; Ṙ_j be the mean gradient of the j-th neighborhood in the i-th local area of Ṙ. The specific loss function can be established as:

${{K}_{LO}}=\frac{1}{L}\sum\limits_{i=1}^{L}{\sum\limits_{j\in M\left( i \right)}{{{\left( \left| \left( \bar{\nabla }{{{\hat{R}}}_{i}}-\bar{\nabla }{{{\hat{R}}}_{j}} \right) \right|-\left| \left( \bar{\nabla }{{R}_{i}}-\bar{\nabla }{{R}_{j}} \right) \right| \right)}^{2}}+{{\mu }_{2}}\times {{\left| \nabla \hat{R}-\nabla R \right|}_{1}}}}$     (19)

Let M be the number of randomly selected non-overlapping local areas; B_l be the mean pixel value of the l-th local area; O be the exposure level. Then, the exposure control loss function can be expressed as:

${{K}_{EC}}=\frac{1}{M}\sum\limits_{l=1}^{M}{\left| {{B}_{l}}-O \right|}$      (20)

To prevent color distortion in the enhanced image, it is necessary to ensure that the ratio of the mean pixel values B of different channels in the enhanced image is consistent to that of the original low-illumination image. Suppose t and w belong to one of the three channels in the RGB color space. Then, the color consistency loss can be calculated by:

${{K}_{CC}}=\sum{_{\left( t,w \right)\in \theta }}{{\left( {{B}^{t}}-{{B}^{w}} \right)}^{2}},\theta =\left\{ \left( S,H \right),\left( S,Y \right)\left( Y,H \right) \right\}$      (21)

The mean squared error loss can be calculated by:

${{K}_{JF}}=\frac{1}{M}{{\sum\limits_{i=1}^{M}{\left( {{b}_{i}}-{{f}_{\omega }}\left( {{a}_{i}} \right) \right)}}^{2}}$      (22)

When humans compare the similarity of two images, they are inclined to focused on the similarity of the spatial structures of the two images. Let λ_a and λ_b be the mean brightness of images a and b, respectively; η_A and η_B be the contrasts of images a and b, respectively; τ₁ and τ₂ be two constants used to prevent the denominator from being zero; D be an extremely small value; δ be the range of pixel values. Then, we have:

${{K}_{JG}}\left( a,b \right)=\frac{\left( 2{{\lambda }_{a}}{{\lambda }_{b}}+{{\tau }_{1}} \right)\left( 2{{\eta }_{a}}{{\eta }_{b}}+{{\tau }_{2}} \right)}{\left( \lambda _{a}^{2}\lambda _{b}^{2}+{{\tau }_{1}} \right)\left( \eta _{a}^{2}\eta _{b}^{2}+{{\tau }_{2}} \right)}$     (23)

${{\lambda }_{a}}=\frac{1}{M-1}\sum\limits_{i=1}^{M}{{{a}_{i}}},{{\eta }_{a}}=\sqrt{\frac{1}{M-1}{{\sum\limits_{1}^{M}{\left( {{a}_{i}}-{{\lambda }_{a}} \right)}}^{2}}}$     (24)

${{\tau }_{1}}={{\left( {{D}_{1}}\delta  \right)}^{2}},{{\tau }_{2}}={{\left( {{D}_{2}}\delta  \right)}^{2}}$     (25)

Let a and d be the number of pixels, and number of channels of the image, respectively; ∇_a and ∇_b be the horizontal and vertical gradients, respectively; θ^t_a,d and θ^t_b,d be the weights of the horizontal and vertical gradients, respectively. Based on prior knowledge, this paper designs a smooth loss function:

${{K}_{PH}}=\sum\limits_{o}{\sum\limits_{d}{\theta _{a,d}^{o}\left( {{\nabla }_{a}}{{R}_{o}} \right)_{c}^{2}}}+\theta _{b,d}^{o}\left( {{\nabla }_{b}}{{R}_{o}} \right)_{d}^{2}$     (26)

Based on deep learning and illumination compensation, this paper attempts to develop an effective a multi-scale image enhancement algorithm. Firstly, a compensation algorithm was developed for image illumination drift based on the Hodge decomposition model. Then, the lightweight multi-scale Retinex network was selected to enhance low-illumination multi-scale images, and the network architecture was presented clearly. In addition, a new joint network loss function was designed for the input and output forms, corresponding to the proposed deep learning network, and the low-illumination multi-scale image enhancement. Through experiments, the authors summarized the linear regression coefficients of low-illumination image database before and after illumination compensation, revealing that our algorithm has the best illuminance compensation effect. The authors also compared the image retrieval correctness of low-illumination image database before and after illumination compensation, and demonstrated that Hodge decomposition-based illumination compensation can effectively enhance the accuracy of image identification and retrieval. Moreover, the change curve of image query correctness with the number of iterations was obtained, and multiple models were quantitatively compared on different low-illumination image datasets. The comparison shows that the combination between lightweight multi-scale Retinex network and Hodge can output images with rich details and bright colors, which are in line with human visual perception and aesthetics. Furthermore, the results of the ablation test shows that the joint use of all seven loss terms led to the optimal effect. Finally, the image enhancement effects were discussed at different scales. The results show that our algorithm has the best image enhancement effect at the scale of 3.