Recent Advances in Image Super-Resolution Reconstruction Based on Machine Learning

Medical images typically contain multiple different regions, some of which are crucial for diagnosis and require extremely high clarity and accuracy. Other regions may be relatively simple, and lower resolution does not affect diagnostic performance. Remote sensing images, which usually come from satellites or drones, have extremely complex surface features, including various textures such as urban areas, forests, lakes, and mountains. The resolution requirements for these regions vary. For example, urban areas may need precise reconstruction of building and road details, while natural landscapes such as forests and lakes may have lower resolution requirements. For these types of images, traditional SR methods often fail to flexibly target the needs of different regions, which may lead to wasted computational resources or imbalanced reconstruction quality. To address this, this paper proposes a new framework for image SR reconstruction with dynamic region selection. The core innovation of this framework lies in the use of reinforcement learning to dynamically select the most suitable SR model for each texture region, rather than simply applying a fixed SR algorithm to the entire image.

This method is effectively modeled through a reinforcement learning framework, where MDP provides a clear theoretical structure. The texture characteristics of different organs or tissues in medical images often vary significantly, and the texture and scale variations of different ground objects in remote sensing images are also very complex. In this framework, the input low-resolution image is divided into several sub-blocks through a segmentation algorithm, each containing different texture features. The goal is to select the most suitable model from a predefined pool of SR models based on these features and apply it to each image sub-block. This approach flexibly addresses the regional heterogeneity in images such as those from medical imaging and remote sensing, improving the accuracy and application of SR reconstruction. By modeling with MDP, the image SR task can be treated as a sequential decision problem, where each state represents the current state of the image, each action is the selection of the SR model to be applied to a specific image sub-block, and the reward function provides feedback based on the MSE between the region’s SR reconstruction and the Groundtruth image.

To allow the reinforcement learning model to dynamically select the most suitable SR model between different image sub-blocks, the action space of MDP is defined as a set of J-dimensional one-hot vectors. In MDP, each image sub-block is a state, and the action selection is determined based on the specific needs and features of each state. The ultimate goal is to optimize the policy so that the overall SR reconstruction effect of the image is optimal. Whenever a specific SR model is chosen to process a particular image sub-block, this action has an immediate effect on a portion of the image, and this effect is fed back to the agent via the reward function. During training, the agent aims to maximize long-term rewards based on the current state and action selection. This requires considering not only the effect of processing the current image sub-block but also taking into account the final reconstruction quality of the entire image. Through this dynamic selection mechanism, MDP can effectively select the best SR model for each image sub-block, thereby improving the overall effect of the SR reconstruction.

1.png

Figure 1. Example of image state transition

In the dynamic selection modeling of the SR model in this paper, the state of MDP determines how the model dynamically selects SR models between different image sub-blocks. To ensure that each state maintains physical significance during the state transition process, the state needs to contain both global and local information of the image. Specifically, in this implementation, the state is defined as consisting of two parts: on one hand, the entire image processed by the SR model, i.e., the image processed in the current iteration; on the other hand, the segmentation label corresponding to the current image sub-block, which defines the image region currently being processed. During the training process, the state is formed by concatenating these two elements, ensuring that the state contains both the global information of the entire image and the local information of the current image sub-block. With this state definition, the state transition in MDP is effectively guaranteed. In the MDP framework, every time an action is performed, the image and state will change. For example, when the agent selects a certain SR model to process the current image sub-block, the image state will change, and the entire image will transition from the current state to the next state, i.e., the current image sub-block is processed and updated to the next image sub-block. This process ensures that the state transitions are meaningful, not independent, and unfold gradually as image sub-blocks are processed. After each iteration, the new state will include the updated image information U^s and the segmentation label L_s of the next image sub-block, thereby maintaining the continuous connection between global information and local information. This state transition method ensures that the reinforcement learning model can maintain a dynamic balance between global and local information throughout the training process, enabling the selected SR model to provide the best reconstruction effect for each sub-block and optimize the final SR image quality through continuous state transitions. Figure 1 provides an example of image state transition. The current state is defined as:

${{T}_{s}}={{U}^{s}}\oplus {{L}_{s}}$             (1)

Assuming the negation of L_s is represented as L^-_s, the SR image after processing the s-th image sub-block using the action x_s is represented by d(U^s|), and pixel-level multiplication is denoted by *; the state transition from U^s to U^s+1 can be represented as:

${{U}^{s+1}}=\overline{{{L}_{s}}}*{{U}^{s}}+{{L}_{s}}*d\left( {{U}^{s}}|{{x}_{s}} \right)$            (2)

Based on the above two formulas, the state transition definition is as follows:

${{T}_{s+1}}=\left[ {{\overline{L}}_{s}}*{{U}^{s}}+{{L}_{s}}*d\left( {{U}^{s}}|{{x}_{s}} \right) \right]\oplus {{L}_{s+1}}$         (3)

In the dynamic decision-making framework based on reinforcement learning, the reward function evaluates the effect of each action to provide feedback on the quality of the SR model selected by the agent, thus optimizing the policy. Specifically, for the image SR task, although the agent does not know which SR model is optimal for each image sub-block at each iteration, by designing a suitable reward function, the agent can gradually learn to select the most suitable model. The reward function proposed in this paper mainly evaluates the reconstruction quality of the current image region and the overall image, with the goal of pushing the agent to select the best SR model by measuring the reconstruction effect of each image sub-block. To achieve this goal, the regional reward function E_Q(U^s,x_s) is designed to measure the difference between the image region processed by the current SR model and the original high-resolution image, with the commonly used metric being the MSE. Mathematically, assuming the standard reinforcement learning reward value is denoted by λ, the threshold is denoted by ς, the Groundtruth high-resolution image is denoted by U_GE, and the image width and height of the target high-resolution image are denoted by Q and G, respectively, E_Q is defined as:

$\begin{aligned} & E_Q\left(U^s, x_s\right)=\begin{cases}\lambda & \frac{1}{Q G}\left\|U^{G E}-d\left(U^{s-1} \mid x_s\right)\right\|_2<\varsigma \\ -\lambda & otherwise \end{cases} \end{aligned}$            (4)

2.png

Figure 2. DQN framework

In the SR task, the core goal of the dynamic policy selection framework based on reinforcement learning is to implement dynamic selection of SR models for different image regions through the DQN. The policy in the framework, i.e., the agent, will select the appropriate action, i.e., the SR model, to process the current image sub-block based on the currently observed state. Specifically, the DQN network continuously receives feedback from the environment, i.e., the reward function, and in each iteration, selects the optimal SR model based on the current state, thereby optimizing the overall SR reconstruction quality of the image. In each round of training, the DQN network will, during the processing of each image sub-block, execute an action to select a pre-trained SR model from the model pool, and update the network weights based on the image reconstruction effect produced by the currently selected model. Figure 2 shows the structure of the DQN used.

In the training phase of the DQN, the agent selects the appropriate action x_s based on observing the current state T_s=U^s⨁L_s, and performs dynamic selection of the SR model. According to the ϵ-greedy strategy, the agent randomly selects an action with probability ϵ, which ensures the exploration of more action space, preventing the model from converging to a local optimum during the early stages of training when the network parameters are not fully refined; while with probability 1-ϵ, the agent selects the optimal action predicted by the DQN network, which enhances the model's exploitation ability. After selecting the action, the agent calculates the reward of the current action based on the regional reward function E(D, X) designed in this paper, and then updates the Q-value function in the Q-network based on this feedback. This process updates the parameters based on the Bellman equation, and through Q-value optimization, the DQN gradually learns how to select the most suitable SR model for each image region.

The Q-learning network structure consists of three convolutional layers and two fully connected layers. Each convolutional layer uses the ReLU activation function, which helps introduce non-linear features and allows the network to effectively learn complex image features. In the convolution operation, the kernel sizes are 8×8, 4×4, and 3×3, aimed at capturing local information at different scales. By using max-pooling layers for dimensionality reduction, the network is able to compress the input feature map in the first two layers, retaining key information while reducing computation. After each action selection and reward acquisition, the DQN network calculates the current Q-value based on the current state T_s and selected action x_s, and compares it with the target Q-value obtained from the environment feedback. Let the discount factor be represented by η, the probability distribution of the state T_s be represented by γ, and the probability distribution of the action be represented by ϑ(·). The target Q-value is estimated using the next state T_s+1 and the maximum Q-value of the selected action, i.e.:

$W^*(t, x)=R_{t \sim r}\left[e_s+\eta M A X_x^{\prime} W^*\left(t^{\prime}, x^{\prime}\right) \mid t, x\right]$            (5)

${{M}_{s}}\left( {{\varphi }_{s}} \right)={{R}_{t,x\sim\vartheta \left( \cdot  \right)}}\left[ {{\left( {{b}_{s}}-W\left( T,x;{{\varphi }_{s}} \right) \right)}^{2}} \right]$            (6)

$b_s=\left\{\begin{array}{l}e_s \quad FO MIN ALS S_{s+1} \\ e_s+\varepsilon M A X_x^{\prime} W\left(t_{s+1}, x^{\prime} ; \varphi\right)  otherwise \end{array}\right.$            (7)

The loss function of the Q-network is:

$\nabla_{\varphi_s} M_s\left(\varphi_s\right)=R_{t s, x \sim \vartheta\left(() ; t_s \sim \gamma\right.}\left[\begin{array}{l}\left(e_s+\eta \underset{x_s^{\prime}}{M A X W}\left(t_s^{\prime}, x_s^{\prime} ; \varphi_{s-1}\right)-W\left(t_s, x_s ; \varphi_s\right)\right) \\ \nabla_{\varphi_s} W\left(t_s, x_s ; \varphi_s\right)\end{array}\right]$           (8)

Table 1 shows the comparison of reconstruction quality before and after the improvement, based on different SR scales (×2, ×3, ×4) for the four datasets: Urban100, T91, DIV2K, and Flickr2K. The evaluation metrics include PSNR and IFC (Structural Similarity Index). From the data before improvement, the PSNR values for the four datasets at the ×2 scale are 36.25, 32.15, 31.25, and 32.24, and the corresponding IFC values are 8.569, 8.235, 7.785, and 8.785. At the ×3 scale, the PSNR values decrease to 32.15, 31.25, 27.88, and 26.35, with the IFC values dropping to 5.265, 4.562, 4.125, and 5.231. The results at the ×4 scale further degrade. These results indicate that without the improved reward function, the image quality sharply declines as the SR scale increases, particularly for datasets 3 and 4, where both PSNR and IFC values show significant degradation. In contrast, the algorithm with the improved reward function demonstrates a noticeable quality improvement, especially at the ×3 and ×4 scales. For example, at the ×3 scale, the PSNR for dataset 1 increases from 32.15 to 33.24, and the IFC improves from 5.265 to 5.214. For dataset 2, the PSNR increases from 31.25 to 31.25, and the IFC improves from 4.562 to 4.568. Additionally, although the improvement for datasets 3 and 4 is relatively weaker, the PSNR and IFC values still show slight improvements, especially at the ×4 scale.

The experimental results show that the proposed improved reward function significantly enhances the image SR reconstruction quality, especially for large-scale upscaling (×3 and ×4), where the effect is more pronounced. Specifically, the improved inter-region reward function effectively alleviates the local inconsistency issue in model selection, enhancing the visual coherence of the image reconstruction by improving the consistency of model selection between adjacent regions. PSNR and IFC values for datasets 1 and 2 show stable and significant improvements at various scales, indicating that the improved algorithm can enhance overall image quality while preserving detail. For datasets 3 and 4, although the improvement is relatively modest, the increase in PSNR and IFC values still demonstrates the adaptability and effectiveness of the improvement method across different types of images.

Based on the experimental data from Tables 2, 3, and 4, significant differences in the performance (such as PSNR and SSIM) and model complexity (such as parameter count and Multi-Adds) of different algorithms are observed for the SR reconstruction task. Firstly, regarding performance metrics, EDSR and RCAN show consistent results across scales ×2, ×3, and ×4, achieving high PSNR values (around 31.26) and SSIM values (ranging from 0.9125 to 0.9245). This indicates that these models are effective in recovering image details and maintaining visual quality. In contrast, while SRCNN and VDSR perform reasonably well at scale ×2 (PSNRs of 28.65 and 28.97, SSIMs of 0.8895 and 0.9125), their performance declines significantly as the scaling factor increases. Especially at scales ×3 and ×4, the PSNR values drop markedly, indicating that these algorithms struggle with higher magnification compared to other deep learning methods. Secondly, in terms of model complexity, the proposed method is relatively optimized in terms of parameter count and Multi-Adds compared to other common SR algorithms (like SRCNN, VDSR, and FSRCNN). For instance, at scale ×2, SRCNN and VDSR have parameter counts of 8K and 12K, while the proposed method has 558K parameters, which is a moderate increase compared to traditional algorithms. However, for scales ×3 and ×4, the proposed method's parameter count is reasonably controlled, being notably lower than EDSR (536K and 546K at scales ×3 and ×4, respectively). Additionally, the proposed method shows good optimization in terms of Multi-Adds. For scales ×3 and ×4, the proposed method has Multi-Adds of 53.2G and 32.6G, respectively, which is lower than traditional deep networks like FSRCNN (72.3G) and EDSR (41.2G). This demonstrates the computational efficiency advantage of the proposed model.

Table 1. Comparison of reconstruction quality with the proposed algorithm before and after reward function improvement

Table 2. Performance and model complexity comparison of different algorithms at scale ×2

Table 3. Performance and model complexity comparison of different algorithms at scale ×3

Table 4. Performance and model complexity comparison of different algorithms at scale ×4

4a.png

4b.png

4c.png

Figure 4. Curves showing the change in PSNR with varying numbers of sub-image regions for different magnification factors

The experimental results suggest that the proposed method outperforms traditional SRCNN and VDSR methods, particularly at higher magnification (×3 and ×4). It strikes a better balance between performance and complexity. Specifically, at scales ×3 and ×4, the proposed method achieves PSNR and SSIM values close to those of more complex models like RCAN and EDSR, but with significantly lower computational complexity. This indicates that the proposed SR framework, based on MDP and reinforcement learning, provides a practical solution for dynamically selecting models and optimizing the reconstruction process, particularly for adaptive handling of different image regions. It ensures high image quality while reducing computational load.

According to the results provided in Figure 4, as the magnification factor increases (×2, ×3, and ×4), the PSNR value gradually improves with an increase in the number of sub-image regions. For a magnification factor of 2, as the number of sub-image regions increases from 3 to 100, the PSNR value increases from 33.2025 to 33.362, indicating a noticeable improvement in image reconstruction quality with finer region segmentation. Similarly, for a magnification factor of 3, the PSNR value increases from 29.995 with 3 regions to 30.108 with 198 regions, further reflecting the impact of region division on reconstruction performance. A similar trend is observed at a magnification factor of 4, where the PSNR value starts at 28.5055 and gradually rises to 28.68. Although the PSNR values differ across magnification factors, in each case, as the number of regions increases, the PSNR value shows a gradual optimization trend. This trend aligns with the SR reconstruction framework proposed in this paper, which is based on MDP and machine learning. Specifically, the introduction of reinforcement learning enables the model to dynamically select appropriate reconstruction methods based on different regions of the image, significantly improving SR reconstruction quality. As the number of sub-image regions increases, the model can optimize smaller regions, improving the accuracy of image reconstruction, especially in complex regions with fine details.

From the experimental results, it can be seen that the proposed dynamic model selection method based on reinforcement learning effectively improves the image reconstruction quality at different magnification factors by increasing the number of sub-image regions. Through the improvement of the reward function between regions, the proposed method can make more precise model selections within local regions, reducing the local inconsistency issues commonly found in traditional SR models. As the number of regions increases, the PSNR value gradually improves, indicating that the consistency of model selection between adjacent image regions has been optimized, and the visual coherence of the image reconstruction has been ensured.