A Hybrid Multiscale and CNN-Based Fusion Framework for Enhanced SAR–MS Image Integration

In general, the high-resolution satellite images get captured with a long revisit time, while low-resolution satellites revisit frequently. Various applications need satellite images with high spatial as well as spectral resolution [1]. Satellite images can be used for fishing prediction, crop species classification, soil analysis, Land Use Land Cover (LULC) changes, biomass and mineralogy mapping, aerospace and shipbuilding, to name a few. This rich information dataset can be obtained by the combination of two or more dissimilar images of same scene, to form new image with an improved quality and reliability [2, 3]. To obtain such usable dataset satellite image fusion of multitemporal and multiresolution images have been researched in recent decades.

1.1 SAR image

Synthetic Aperture Radar (SAR) is an active microwave sensor that utilizes long-wavelength electromagnetic radiations, enabling it to effectively penetrate atmospheric impairments such as cloud cover, haze, dust, fog, smoke and other adverse environmental factors with the exception of intense penetration. This characteristic allows SAR imagery to be captured continuously, regardless of the time of day or weather conditions. This mechanism makes SAR imagery richer in spatial information.

1.2 Multispectral image

Multispectral sensors are passive remote sensing instruments that capture solar radiation reflected from the Earth's surface across different parts of the electromagnetic spectrum, such as ultraviolet, visible, and infrared wavelengths. Depending on their spectral resolution, optical sensors are categorized into panchromatic, multispectral and hyperspectral types. Multispectral imaging provides extensive information regarding the spectral signatures of terrestrial objects, thereby facilitating the differentiation of various land cover types. However, it’s effectiveness is significantly influenced by solar illumination and prevailing weather conditions. As these sensors exhibit high spectral sensitivity, it is more readable than SAR images. Therefore, fusion of SAR and multispectral image can provide more informative image with respect to spatial and spectral information. Broadly, multispectral and SAR image fusion is categorised into three different levels: pixel level, decision level, and feature level as per the different levels of information fusion [4-6]. This article focuses on pixel level fusion, offering a more comprehensive discussion on this technique than other methods. Pixel level fusion attains more precise and detailed information, facilitating enhanced interpretation and broader application possibilities. Pixel-level fusion methods can be sorted into four main groups:

1) Component substitution (CS)

2) Multi-scale transform (MST)

3) Model optimization (MO)

4) Hybrid methods [7]

The working principle of CS methods is to transform the multispectral image into a different domain to separate their spatial and spectral information. Then the spatial components are replaced with SAR images and an inverse transformation is applied to restore the images to their original domain by completing the image fusion process [8]. The CS methods inject more spatial details into fused image however spectral distortion may occur [9]. CS methods are the most widely utilized techniques which encompasses, Generalized Intensity-Hue-Saturation Adaptive Algorithm (GIHSA) [4], Gram-Schmidt Context Adaptive Sharpening (GSA-CA) [10], Principal Component Analysis (PCA) [11], Brovey Transform (BT) [3, 6, 7]. In previous studies, these approaches have been vastly employed for the panchromatic and multispectral image fusion. Further advancements in satellite technology have facilitated the availability of SAR and MS imagery for research purpose, offering enhanced information content. SAR and multispectral images usually have huge illumination difference because of different capturing mechanism. Therefore, applying CS methods for SAR and multispectral image pairs is challenging as these methods are good at retaining global details than the local details [7].

The MST methods typically decompose SAR and multispectral images into low and high frequency components and designs fusion rules specifically tailored to align with their respective spectral characteristics [12]. Multiscale geometric analysis, wavelet transform (WT) [13] and dual-tree complex wavelet transform (DTCWT) [14] are the mostly used traditional methods in MST. Multiscale geometric analysis involves curvelet transform (CVT) [15], non-subsampled contourlet transform (NSCT) [16], Bandlet transform [16], contourlet transform (CT) [17], Shearlet transform (ST) [18], and Gaussian and Laplacian pyramid decomposition-based methods. Pyramidal transformation and wavelet transformation-based methods are prominent categories within this group of methodologies. Pyramidal techniques decompose source images into Gaussian and Laplacian pyramids. While MST methods exhibit superior performance relative to CS methods by offering enhanced edge information and preserving cures within images, they are nevertheless subject to certain limitations. These include restricted directional information, the presence of blocking artifacts, and a moderate signal-to-noise ratio [19].

The MO method has been adapted for multispectral and SAR image fusion, treating the process as image restoration. This approach involves constructing a relationship model between the input images, optimizing the energy function and generating the fused image by refining the model [19]. MO methods are mainly composed of variational optimization models and sparse representation models, both of which demand prior knowledge, intricate model development, and significant computational resources. In contrast, deep learning techniques have gained considerable attention in recent years.

Deep learning-based fusion strategies can be classified according to their underlying architecture into autoencoder (AE), convolutional neural network (CNN), and generative adversarial network (GAN) approaches [20, 21]. AE methods typically involve pre-training an autoencoder for feature extraction and image reconstruction, with intermediate feature fusion following traditional rules. A notable example is DenseFuse [22], which trains its encoder and decoder on the MS COCO dataset and employs addition and L1-norm fusion strategies. CNN methods incorporate convolutional neural networks into image fusion in two distinct ways. Prabhakar et al. [23] utilize carefully designed loss functions and network structures to perform feature extraction, fusion, and image reconstruction in an end-to-end manner. Zhang et al. [24] proposed a proportional maintenance loss of gradient and intensity to guide direct fused image generation. The trained CNN-based approaches can formulate the fusion rules while relying on traditional methods for image reconstruction [20, 21, 23-25]. Lian et al. [26] utilized CNN to generate fusion weights, with image decomposition and reconstruction handled by Laplacian pyramids. GAN methods leverage the adversarial interplay between the generator unit and discriminator unit to estimate target probability distributions, implicitly accomplishing feature extraction, fusion, and image reconstruction. FusionGAN pioneered GAN-based image fusion, establishing a relation between the fused and visible images to enhance texture details in the resultant fused image [27-29]. Owing to the considerable differences among image fusion tasks, these techniques are applied in distinct ways across various fusion contexts. In summary, the review highlights important challenges associated with state-of-the-art methods:

1. Spectral distortion continues to be an issue due to the fusion of heterogeneous images.
2. Handcrafted fusion rules lack adaptability.
3. The interpretability of deep learning fusion remains limited.

The lacunas in state-of-art fusion models can be minimised by the hybrid method implementation. This research work is synthesized with fusion and reconstruction of detail, approximate and contour layers of input images. The key contributions of the implementation are:

1. Computed contour layer using Rolling Guidance Filter (RGF) based image decomposition.
2. Effective fusion rule designed for individual layer fusion.
3. Dynamic weight computation using augmented dataset training of CNN for SAR and MS image fusion.
4. Integration of enhanced spatial and spectral information using multiscale decomposed and CNN based fusion.

The research work is presented as follows: Section 2 introduces the proposed methodology, offering a succinct comparison with state-of-the-art techniques. The following Section 3 elaborates on details of the experimental design, data and the evaluation indexes used. Section 4 discusses the results and provides an analysis. The final section concludes with a synthesis of the principal findings.

1. Experimental data

In the course of rigorous experimentation, the MS image dataset employed was procured from the National Remote Sensing Centre (NRSC) in Hyderabad, India. Conversely, the SAR image dataset was sourced from the Earth Resources Observation and Science (EROS) Centre of the U.S. Geological Survey via the freely accessible QGIS tool 3.18 version. This dataset encompasses the regions within the state of Maharashtra, India. The SAR image possesses a resolution of 15 meter, whereas the LISS III Multi Spectral image exhibits a resolution of 8 meter. The dataset used consists of more than 100 MS and SAR image pairs, in this presentation results of pairs is discussed. Prior to initiating the fusion process, an initial pairwise registration is conducted to achieve optimal results. The experimental setup used is CPU Intel Core i7-12500 H_z 12th gen 2.50 GHZ, 2GB GPU, 64 bit operating system Windows 11, Programming environment PyCharm 2024.3.

2. Evaluation indexes

Qualitative evaluation entails a subjective analysis of outcomes by comparing texture details, color information, spatial structure, visual effects, and other features of the combined images. On the other hand, quantitative evaluation offers an objective analysis based on specific evaluation metrics.

In this section, we report the experimental process for diverse satellite image dataset using proposed algorithm. The experiments on four SAR and MS image datasets to validate the proposed fusion algorithm. For the purpose of bench marking, we chosen the listed baseline methods in the experimentations as, NSCT, PCA, Brovey Transform (BT), DWT [50], FusionGAN [29], and Siamese Network [51]. The experiments compare fused image with the reference image to analyse the performance of proposed method. Here, we utilize Peak Signal to Noise Ratio (PSNR), Spectral Angle Mapper (SAM), Erreur Relative Globale Adimensionnelle de Synthese (ERGAS), Average Gradient (AG), and Information Entropy five measures for proposed method evaluation. PSNR is a measure of accuracy of an algorithm. SAM signifies the spectral distortion after fusion. ERGAS indicates radiometric and spatial quality of an image.

1. PSNR

This is crucial metric for measuring the quality of an image after processing. It evaluates the ratio of the maximum achievable power of an input signal to power of the noise, defined using Eq. (19).

$P S N R=10 \times \log _{10} \frac{255}{M S E}$                     (19)

Here, MSE denotes mean square error of fused image as explained in Eq. (20). Let’s consider I_f(x_k’, y_k’) denote the transformed coordinates of I_a(x_k, y_k) from original image.

$M S E=\frac{1}{N} \sum_{k=1}^N\left(I_a\left(x_k, y_k\right)-I_f\left(x_k^{\prime}, y_k^{\prime}\right)\right)^2$                 (20)

Here, N represents the number of difference pairs.  I_a, I_f indicates input and fused image respectively.

2. SAM

The spectral quality is evaluated using SAM, it typically compares pixel-wise spectral similarity of fused image with reference image by computing angle between two vectors. Eq. (21) evaluates spectral similarity between K and T, fused and reference image spectral pixel vectors respectively.

$\operatorname{SAM}(K, T)=\cos ^{-1} \frac{(K \times T)}{\|K\|\|T\|}$                       (21)

3. ERGAS

The ERGAS is used for global relative error computation, as shown in Eqs. (22) and (23) [7].

$E R G A S=100 \frac{a}{b} \sqrt{\frac{1}{P} \sum_{P=1}^P\left(\frac{R M S E\left(I_a, I_f\right)}{\mu I_a}\right)}$                     (22)

$\frac{1}{N M} \sqrt{\sum_{i=1}^N \sum_{j=1}^M\left(I_a\left(x_k, y_k\right)-I_f\left(x_k^{\prime}, y_k^{\prime}\right)\right)}$                        (23)

where, a/b is ratio of resolution of SAR and MS image, P denotes the number of bands and $\mu$ is mean value of the image.

4. AG

This signifies the average magnitude of the image gradient, computed using discrete derivatives as follows

$A G=\frac{1}{(M-1)(N-1)} \sum_{i=1}^{M-1} \sum_{j=1}^{N-1} \sqrt{\left(\frac{\partial f}{\partial x}\right)^2+\left(\frac{\partial f}{\partial y}\right)^2}$                       (24)

Here, M, N is image dimensions, f(i,j) is pixel intensity.

$\begin{aligned} & \frac{\partial f}{\partial x}=f(i+1, j)-f(i, j) \\ & \frac{\partial f}{\partial y}=f(i, j+1)-f(i, j)\end{aligned}$

This measures the strength of edges and textures, averaged throughout the image [49].

5. Entropy

This parameter gives idea about the richness of information in the data under consideration, calculated using Eq. (25). The higher the entropy value, better the quality of fused image [4].

$e=-\sum_{n=1}^{M-1} p(n) \log _2 p(n)$                       (25)

where, p(n) is probability of occurrence of n^th gray level, M is the dynamic range of the image under analysis.

6. Standard Deviation of difference image

This calculation determines the Standard Deviation (SD) of residual information, indicating the extent to which the fusion process has altered the input image. The absolute difference between the source and fused image is initially calculated using Eq. (26), and this result is then used to determine the SD_diff using Eq. (27).

$I_{\text {diff}}=\left|I_{\text {source}}-I_{\text {fused}}\right|$                     (26)

$S D_{d i f f}=\sqrt{\frac{1}{N-1} \sum_{i=1}^N\left(I_{d i f f}(i)-\mu_{d i f f}\right)^2}$                       (27)

where, $I_{\text {diff}}$ is absolute difference image, $\mu_{\text {diff}}$ is mean of difference pixel image and $N$ is total number of pixels.

Despite the higher computational demands of pixel-level fusion techniques compared to feature-level and decision-level fusion methods, they continue to be widely used in remote sensing image fusion due to their superior accuracy. In this article pixel level fusion of SAR and MS image is carried out using RGF multiscale decomposition in combination with CNN based fusion. In the initial phase of fusion, the Intensity (I) component of the multispectral (MS) image is predominantly considered, as it encapsulates the most essential perceptual information. Modifying the I component facilitates the effective integration of spatial details, particularly from the SAR image, thereby enhancing the spatial richness of the fused image. Processing of the I component allow for more controlled enhancement while preserving spectral integrity, and any distortions introduced are relatively easier to rectify during the inverse transformation process. RGF based multiscale decomposition can reserve edge information better than other decomposition methods such as wavelet-based decomposition. After decomposition approximation layer, detail layer and contour layers are obtained and fused individually. Approximation layer is fused using WLE and WSEML methods to obtain most of the information from both input images. WLE method gives advantage of analysing frequency band-wise better to keep spectral information intact and WSEML method primarily focuses on improving the entropy and mean of Laplacian is responsible to ensure spectral consistency. The fusion of the contour layer is executed using a Local Statistical Edge Model, which is designed by taking into account both standard deviation and gradient. This approach aids in preserving crucial edge information, particularly in relation to directional data. Whereas, the detail layer fusion utilises absolute maximization to preserve high frequency information. Subsequently, the average of all layers is integrated with the fusion output from the CNN approach. In CNN-based fusion, the network is trained using the SSIM loss function and augmented dataset pairs of SAR and MS images, which enhances the network's robustness to illumination variations and maintains a balance in smoothness that promotes the preservation of fine details. Through this the weights are calculated and image fusion is accomplished which has ability to find and preserve complex features from dissimilar datasets. By integrating multiscale decomposition with CNN-based fusion methods, the resulting image is enriched with both spatial and spectral information. Following this, the modified I component is processed through an inverse IHS transformation. This innovative method demonstrates superior performance compared to both traditional and network-based techniques. It achieves a peak PSNR of 34 and a minimum SAM of 0.54, indicating significant improvements in the spatial and spectral resolution of the final image. Also ensures high fidelity preservation by maintaining the standard deviation of different images at a low value, ranging from 12 to 19.