Super-Resolution Reconstruction Method Integrated with Image Registration

Super-Resolution Reconstruction Method Integrated with Image Registration

Ling Tang 

Sichuan University Of Science & Engineering, Zigong 643000, China

Corresponding Author Email:
| |
| | Citation



In this paper, a new super-resolution (SR) reconstruction method integrated with image registration using Maximum a Posterior (MAP) formulation is proposed, which combines image registration and reconstruction based on the formulation in the loop iterations. In order to solve the problems of inaccurate registration parameters, a self-adapting weight coefficient is introduced in the cost function of each LR image to allow for continual adjusting. Results of experiments show that the proposed algorithm can not only significantly improve the accuracy of registration, but also effectively improve the reconstruction quality.


image registration, super resolution reconstruction, MAP, self-adapting weight coefficient

1. Introduction

Super-resolution(SR) reconstruction produces a high-resolution(HR) image from warped, blurred and noisy multiple low-resolution(LR) images[1][2]. Reconstruction algorithm generally consists of three parts: image registration (or motion estimation), interpolation, and blur and noise elimination. There are also many spatial domain algorithms which combine interpolation with blur and noise elimination, such as projection onto convex sets, maximum a posterior, etc. However, throughout the whole process image registration is the first and key step to achieve super-resolution reconstruction.

Since the sampling frequency of the low-resolution observation sequences is below the Nyquist sampling frequency[3], low-resolution image registration error exists because of image aliasing, resulting in the appearance of the combined algorithms of registration and reconstruction. Yu He et al[4] presents a nonlinear least squares iterative algorithm, making registration simultaneous in super-resolution reconstruction, and achieved good results. There are some variables based on projection methods which have also been applied for reconstruction, such as Chung [5] who presents a nonlinear cost function using Gauss-Newton method for image registration and reconstruction, as well methods based on statistical probability theory [6] which have also been widely used, and so on [7][8].

However, these above algorithms do not take into account that every LR plays a different role on the HR, and errors in registration parameters exist in each iteration[9]. This paper proposes an adaptive weighted iterative algorithm using self-adapting weight coefficient in the cost function of each LR image. With continuous adaptive iterative updates, the method can achieve more accurate results.

2. Observation Model

First we illustrate the observation model of the original high-resolution image and the actual low-resolution images, as shown in Figure 1.

 Figure 1. Observation model of super-resolution reconstruction

Consider p LR images each of size MxM written in lexicographical notation as the vector yk for k=1,2,...p,z is the desired HR image. From Figure 1, the observed LR images result from warping, blurring and subsampling operators performed on the HR image z. Thus the observation model is:

$y_{k}=D_{k} B_{k} M_{k} z+n_{k}$     (1)

Where D is a subsampling matrix, Mk represents a warp matrix, Bk is a blur matrix, and nk represents a lexicographically ordered noise vector. D, B defaults to the known quantities[10]. In general super-resolution applications, B is calculated by the point diffusion equation of the optical system, with a two-dimensional Gaussian distribution.

3. Algorithm

3.1 Combined solution model

Let the full set of LR images be denoted as $y=\left\{y_{1}, y_{2}, \cdots, y_{p}\right\}$, and the set of motion parameters be denoted as $m=\left\{m_{1}, m_{2}, \cdots, m_{p}\right\}$. The objective of this paper is solving the registration parameters m and high-resolution image z, while at the same time knowing the observation sequences of low-resolution images. According to the maximum a posteriori estimation theory[11] there is:

$\hat{z}, \hat{m}=\arg \max \{p(z, m / y)\}$            (2)

According to Bayes’ rule and taking into account that m and y are independent, we can get:

$\hat{z}, \hat{m}=\arg \max \{p(y \mid z, m) p(z) p(m)\}$          (3)

Both sides of the equation were taken as the logarithm:

$\hat{z}, \hat{m}=\arg \max \{\log p(y \mid z, m)+\log p(z)+\log p(m)\}$   (4)

As seen from the above equation, three probability density functions are the key to calculate. This article assumes that three probability density functions are in line with the general Gaussian distribution, including:

$p(y \mid z, m)=H_{1} \exp \left\{-\frac{\|y-D B M z\|^{2}}{2 \sigma^{2}}\right\}$          (5)

$p(z)=H_{2} \exp \left\{-\lambda_{1}\left\|Q_{1} z\right\|^{2}\right\}$       (6)

$p(m)=H_{3} \exp \left\{-\lambda_{2}\left\|Q_{2} m\right\|^{2}\right\}$          (7)

Where, $H_{1}, H_{2}$ and $H_{3}$ are the constants, $\lambda_{1}$ and $\lambda_{2}$ are adjusted parameters, $Q_{1}$ and $Q_{2}$ are stable matrices. In this paper, consider $Q_{1}=1, Q_{2}$ is a two-dimensional Laplacian. Take Eq. $(5)-(7)$ into $(4),$ we come to:

$\hat{z}, \hat{m}=\arg \min \left\{\|y-D B M z\|^{2}+\lambda_{1}\|z\|^{2}+\lambda_{2}\left\|Q_{2} m\right\|^{2}\right\}$      (8)

The impact of each piece of the actual observed low-resolution images is different on the reconstructed SR image. Taking this into account, adaptive weight cis introduced to update it (ck is the weight of the impact of the kth LR image and SR image), which can be more accurately characterized in the image reconstruction model, as follows:

$\hat{z}, \hat{m}=\arg \min \left\{\sum_{k} c_{k}\left\|y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right\|^{2}+\lambda_{1}\|z\|^{2}+\sum_{k} \lambda_{2}\left\|Q_{2} m_{k}\right\|^{2}\right\}$     (9)

So we come to the energy function:

$E(z, m)=\sum_{k}|| c_{k}\left(y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right) ||^{2}+\lambda_{1}\left\| z\right\|^{2}+\sum_{k} \lambda_{2}\left\| Q_{2} m_{k} \right\|^{2}$    (10)

The purpose of this paper is to solve the equation’s minimization problem through the iteration of registration parameters and high resolution images to solve before convergence. If we know the registration parameters m, the item not including z of the above equation can be deleted, and the energy function of z becomes:

$E(z)=\sum_{k}\left\|c_{k}\left(y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right)\right\|^{2}+\lambda_{1}\|z\|^{2}$      (11)

At this point the solution of z is transformed into a regularized optimization problem, which can be solved through a conjugate gradient method.

If knowing the high-resolution image z, we can determine the energy function of the registration parameter m:

$E(m)=\sum_{k}\left\|c_{k}\left(y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right)\right\|^{2}+\sum_{k} \lambda_{2}\left\|Q_{2} m_{k}\right\|^{2}$       (12)

Thus the solution of m is a regularization optimization problem.

In the optimal process above, an initial estimation of registration parameters and high-resolution image are required. Initializing a high resolution image can be achieved by bilinear interpolation to the reference image. Registration parameters can be implemented by common algorithms. Then, gradually updated registration parameters for estimating the high-resolution image are used, which can more enhance the effects of reconstruction.

3.2 Adaptive weight selection

This paper uses weight so that the algorithm can automatically select different weights according to the size of the LR image, and be automatically revised in the iterative process, with a better adaptive effect. We select the linear solution of ck [12], that is:

$c_{k}=\frac{Q}{\left\|y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right\|^{2}}$       (13)

Where  $Q=\frac{L}{\sum_{k=1}^{L} \frac{1}{\left\|y_{k}-D B_{k} M_{k}\left(m_{k}\right) z\right\|^{2}}}$.

Before the ith iteration, $m_{k, i}, z_{i}$ can be insert into the above equation to calculate the adaptive weight $c_{k, i}$. This value is then brought to the algorithm for computing, during iteration the weight will be adjusted depending on the solution, so as to achieve better reconstruction results.

4. Simulations and Analysis

The original high-resolution image selected is shown in Figure 2. A sequence of LR is generated as follows: first shifting the HR image with 0-4 pixels in the horizontal and vertical directions, and blurring using Gaussian function with the variance of 1, then down-sampling the blurred image in the factor of 2, to obtain four different LR images, which are taken into the algorithm for SR reconstruction. To verify the effectiveness of the proposed algorithm, we compare the propose algorithm with nonlinear least squares algorithm (NLS) and the algorithm in literature [6].

Figure 2. The original HR image

Experiment 1 We used Normalized Mean Square Error(NMSE) and Peak Signal-to-Noise Ratio(PSNR) for quantitative evaluation to the registration parameters and the estimated result of HR image.

As seen in the Figure 3, because of the adaptive weight update to reconstruction, which takes into account the different roles of each LR image to the reconstructed HR image, the proposed algorithm’s NMSE values are smaller, thus significantly improving the accuracy of image registration. Furthermore, bigger PSNR values are obtained, which results in better performance of the algorithm and higher quality of the reconstruction.

(a) NMSE curve of registration parameters

(b) PSNR curve of reconstruction image

Figure 3. Reconstruction results under different SNR

Experiment 2 We compared the proposed algorithm with the algorithm in [6] and the algorithm known registration parameters to image sequences reconstruction when SNR=30dB. Figure 4 shows the reconstruction results, in which can be seen the HR image reconstructed by the new algorithm. Regardless of resolution or degree of detail in the image, it is relatively valid, and the effect is closer to the method knowing registration parameters.

(a) Reference LR

(b) Algorithm in literature

(c) The proposed algorithm

(d) Reconstruction knowing registration parameters

Figure 4. Comparison of the reconstructed images

5. Conclusion

A super-resolution reconstruction method integrated with image registration is proposed in this paper. It combines image registration and reconstruction in the loop iterations. Weight coefficient is introduced in the cost function of each LR image for continual adjusting as cost function’s residuals change, which can achieve better reconstruction effects. The simulation results confirm that the new algorithm can significantly improve the accuracy of registration, and effectively improve the quality of reconstruction. It will have beneficial applications in biomedicine, video surveillance and the other fields.


This work was supported by projects of Sichuan Provincial Department of Education (13ZB0138) and projects of Artificial Intelligence Key Laboratory of Sichuan Province (2013RYY02).


1. Babacan S. D., Molina R., Katsaggelos A. K., Variational Bayesian Super Resolution, IEEE Transactions on Image Processing, vol. 20, pp. 984-999, 2011.

2.   LIU Zhe, ZHANG Yong-liang, HAO Min-hui. et al., A Fast Super Resolution Image Reconstruction Algorithm. Journal of Optoelectronics·Laser, vol.24, pp. 372-377, 2013.

3. Costa G. H., Informed Choice of the LMS Parameters in Super Resolution Video Reconstruction Applications, IEEE Transaction on Signal Processing, vol. 56, pp. 555-563, 2008.

4.    HE Y., YAP K. H., CHEN L., et al., A Nonlinear Least Squaretechnique For Simultaneous Image Registration and Super-Resolution, IEEE Transaction on Image Processing, vol. 16, pp. 2830- 2841, 2007

5. CHUNG J, HABER E, NAGY J, Numerical methods for coupled super resolution, Inverse Problem, vol. 22, pp. 1261-1272, 2006

6. Zhang Hong-yan, Shen Huanfeng, Zhang Liangpei, et al. Blind Super-Resolution Reconstruction Method Based on Maximum a Posterior Estimation, Journal of Computer Applications, vol. 31, pp. 1209- 1213, 2011.

7. HARDIE R. C., BARNARD K. J., ARMSTRONG E. E., Joint MAP Registration and High-Resolution Image Estimation Using a Sequence of Undersampled Images, IEEE Transaction on Image Processing, vol.6, pp. 1621-1633, 1997.

8. WOODS N. A., GALATSANOS N. P., KATSAGGELOS A. K., Stochastic Methods for Joint Registration, Restoration, and Interpolation of Multiple Undersampled Images, IEEE Transaction on Image Processing, vol. 15, pp. 201-213, 2006.

9.  S. S. Panda, G. Jena, S. K. Sahu, Image Super Resolution Reconstruction Using Iterative Adaptive Regularization Method and Genetic Algorithm, Computational Intelligence in Data Mining - Volume 2, Springer India, vol. 32, pp. 675-681, 2015.

10.  Sroubek F., Flusser J., A Unified Approach to Super Resolution and Multichannel Blind Deconvolution, IEEE Transactions on Image Processing, vol. 16, pp. 2322-2331, 2007.

11. SCHULTZ R. R., STEVENSON R. I., A Bayesian Approach to Image Expansion for Improved Definition, IEEE Transaction on Image Processing, vol.3, pp. 233-242, 1994.

12. Yang Xin, Fei Shumin, Zhou Dake, Self-Adapting Weighted Technology for Simultaneous Image Registration and Super-Resolution Reconstruction Based on MAP, Chinese Journal of Scientific Instrument, vol. 32, pp.1771-1775, 2011.

13. Xie Kai, Zhang Fen, Efficient Parameter Estimation Algorithm for Super-resolution Image, Journal of Chinese Computer systems, vol. 34, pp.2201-2204, 2013

14. NG M. K., SHEN H., LAM E., et al., A Total Variation Based Super-Resolution Reconstruction Algorithm for Digital video [EB/OL],, May 2010.

15. Toygar Akgün, Murat Gevrekci, Accelerating Super-Resolution Reconstruction Using GPU by CUDA, Advances in Computational Science, Engineering and Information Technology, Springer International Publishing, vol. 225,  pp. 47-58, 2013.