- Research
- Open Access

# Super-resolution via a fast deconvolution with kernel estimation

- Han Yu
^{1}, - Ting-Zhu Huang
^{1}Email authorView ORCID ID profile, - Liang-Jian Deng
^{1}and - Xi-Le Zhao
^{1}

**2017**:3

https://doi.org/10.1186/s13640-016-0125-6

© The Author(s) 2016

**Received:**27 September 2015**Accepted:**11 July 2016**Published:**25 July 2016

## Abstract

Image super-resolution has wide applications in biomedical imaging, computer vision, image recognition, etc. In this paper, we present a fast single-image super-resolution method based on deconvolution strategy. The deconvolution process is implemented via a fast total variation deconvolution (FTVd) method that runs very fast. In particular, due to the inaccuracy of kernel, we utilize an iterative strategy to correct the kernel. The experimental results show that the proposed method can improve image resolution effectively and pick up more image structures. In addition, the speed of the proposed method is fast.

## Keywords

- Super-resolution
- Kernel estimation
- Fast total variation deconvolution (FTVd)

## 1 Introduction

The process of estimating a high-resolution (HR) image from one or multiple low-resolution (LR) images is often referred to as image super-resolution. According to the number of low-resolution images, image super-resolution can be divided into two categories: one is single-image super-resolution, and the other is multiple-image super-resolution. Based on image sequence, multiple-image super-resolution uses overlapping information between multiple low-resolution images to estimate details of the high-resolution image [1–3]. Due to multiple-image super-resolution needs more than one input image, it cannot deal with the situation when only one image is inputted. In this paper, we mainly focus on single-image super-resolution.

Interpolation-based methods are one of classical image super-resolution methods. To determine pixel values of each position in the high-resolution image, interpolation-based methods need to construct a rational interpolation function. The conventional interpolation-based methods contain bicubic interpolation method, cubic interpolation method [4], cubic spline interpolation method, nearest-neighbor interpolation method, etc. These methods usually run very fast but always produce blurring or jagged artifacts. Recently, many contributions in terms of interpolation-based methods have been proposed [5–11]. In [6], Zhang et al. present an edge-guided interpolation algorithm through directional filtering and data fusion. This method can preserve sharp edges. In [9], Wang et al. propose a fast image upsampling method within a two-scale framework. They use interpolation method to recover the low-frequency image and reconstruction technique to recover the local high-frequency structures.

Reconstruction-based methods are another class of image super-resolution approaches. Through studying the low-resolution image generating process, reconstruction-based methods use a certain model to depict the mapping relationship between the high-resolution image and the low-resolution image. There are three main types of reconstruction-based methods: frequency domain techniques, spatial domain techniques, and other techniques. Frequency domain techniques [12–14] solve the problem in frequency domain, and the observation model is based on displacement characteristics of Fourier transform. Spatial domain techniques, such as non-uniform interpolation within samples method [15], convex set projection method [16], statistical recovery method (maximum a posteriori and maximum likelihood estimation) [2, 17, 18], solve the problem in spatial domain. In addition, there are some other reconstruction-based methods [19–24]. In [22], Shan et al. propose an efficient upsampling method which lies in a feedback-control framework. This method runs very fast and can preserve the essential structural information.

Learning-based methods are the third category of image super-resolution methods. Learning-based methods [25–32] need to train two dictionaries for low-resolution and high-resolution images or patches. When given a low-resolution image, learning-based methods can get a high-resolution image through using the learnt mapping relationship between the two dictionaries. Although these methods obtain good visual results, they rely on the two training dictionaries and cannot change the magnification factor arbitrarily.

In this paper, we propose a new single image super-resolution method based on deconvolution strategy. According to Efrat et al. [33], an accurate kernel is more important than a sophisticated prior for image super-resolution. Thus, we take into account the error of blur kernel in our method. We develop an iterative strategy to adjust the blur kernel and then estimate the final high-resolution image via a reconstruction method. The proposed method is based on the framework of [22]. However, it has two main contributions. First, the proposed method can get faster speed than [22], since we employ a fast total variation deconvolution (FTVd) method in our work. Second, the proposed method estimates the deconvolution kernel iteratively to get better visual and quantitative results than [22].

The rest of this paper is organized as follows. Section 2 introduces image super-resolution problem, reviews FTVd method and a fast image upsampling method. In Section 3, we detail the proposed model and algorithm. Numerical results are shown in Section 4. Finally, we draw some conclusions in Section 5.

## 2 Problem description and related works

## 3 The proposed method

### 3.1 The proposed framework

In this section, we give the proposed method which is consisted of four parts: deconvolution, estimating blur kernel, reconvolution, and pixel substitution.

*L*. We first transform the low-resolution image from RGB color space to YUV color space. Next, we upsample the low-resolution image to an ideal size by bicubic interpolation method and only conduct at Y space. We take an iterative strategy to achieve the image upsampling process (see Algorithm 2). Our strategy contains four parts: deconvolution, estimating blur kernel, reconvolution, and pixel substitution. We take deconvolution process to eliminate the effect of the linear filtering. For instance, there are some visual artifacts around the image “wheel” after bicubic interpolation in the Y space (see Fig. 1). Besides, because the accurate blur kernel can not be known exactly, the further estimated high-resolution image will become more inaccurate. Thus, we take account of the error of blur kernel. Furthermore, the same as [22], we take reconvolution and pixel substitution process to control the image upsampling. By applying our strategy iteratively at the initial high-resolution image

*B*

^{(0)}, we can obtained the estimated high-resolution image at Y space. The final estimated high-resolution image

*H*

^{∗}is acquired by transforming the high-resolution image from YUV color space to RGB color space. We will show more details about the four steps of the proposed method (deconvolution, estimating blur kernel, reconvolution, and pixel substitution) as follows.

#### 3.1.1 Deconvolution

Let B ^{(i)} as a high-resolution that is gotten at iteration *i,i*≥0, and *B*
^{(0)} is obtained by bicubic interpolation method. In particular, *B*
^{(0)} is obtained by bicubic interpolation method. The deconvolution process, estimating the high-resolution image *H*
^{(i)}, can be regarded as solving the first part of Eq. (2). This problem can be modeled as ∥*f*⊗*H*−*B*∥, where ⊗ is a convolution operator. Let *W* be a convolution matrix, then ∥*f*⊗*H*−*B*∥ can be regarded as a multiplication of *W* and *H*. Note that the problem of minimizing \(\|f\otimes H- B\|_{2}^{2}\) is hard to solve because the inverse of *W* does not always exist and sometimes *W* can be influenced by noise. In particular, taking the high-resolution image *B* as a blurred image, the deconvolution process can be considered as an image restoration problem. In this paper, we take FTVd method [34] in the deconvolution process, since FTVd method [34] is an effective way to deal with image restoration problem. The main steps of FTVd method are shown in Section 2.1.

#### 3.1.2 Estimating blur kernel

Because the blur kernel is not known exactly in the image formation process, it may have some errors: \(f=\tilde {f}+e\), where *f* is the accurate blur kernel and \(\tilde {f}\) is the inaccurate blur kernel containing an error *e*. In order to get a reasonable high-resolution image, we need to consider the error of blur kernel.

*Ω*∈

*R*

^{2}with compact Lipschitz boundary, we take the initial high-resolution image

*B*

^{(0)}, acquired by bicubic interpolation, as a blurred image and the intermediate high-resolution image at

*i*times

*H*

^{(i)}as a real image. \(\tilde {f}\) be the blur kernel and

*e*be the error of blur kernel. We study the following objective function to get the error of blur kernel

*e*:

*α*is a positive regularization parameter. This problem can be solved by fast Fourier transform:

where \(\mathcal {F}\) and \(\mathcal {F}^{-1}\) are the Fourier transform and the inverse Fourier transform, respectively. When we compute the error *e*, then the blur kernel can be estimated by \(f=\tilde {f}+e\).

#### 3.1.3 Reconvolution

Taking account of reconvoluting the output image *H*
^{(i)} with the blur kernel \(\tilde {f}\), the result should be close to *B*
^{(i−1)}, where *i*≥1. If not, there must be some incorrect pixel values in *B*. So we need to modify the high-resolution image using the low-resolution image information, which leads to pixel substitution in the next step. In particular, in each reconvolution step, we choose the initial blur kernel \(\tilde {f}^{(0)}\) to reconvolute the high-resolution image. If we choose the updated blur kernel in the reconvolution process, the high-resolution image cannot be well estimated due to the change of blur kernel.

*d*. In addition, the subsampling process only keep one pixel in the high-resolution image. Thus, the corresponding pixels in the high-resolution image can be substituted for pixels in the low-resolution image. In this paper, we take the pixel substitution strategy the same as [22] (see Fig. 2). If we upscale the low-resolution image for

*d*times, we use the pixel (

*i,j*) in the low-resolution image

*L*to replace the pixel (

*d*×

*i*+1,

*d*×

*j*+1) in the corresponding high-resolution image

*Reblur*.

Then, we can use the pixel-replaced image to conduct the next iteration. After several iterations, the estimated high-resolution image *H*
^{∗} can be obtained. Our algorithm is given in Algorithm 2.

### 3.2 The difference between [22] and the proposed method

In [22], Shan et al. introduce a fast image/video upsampling method that involves a feedback-control framework. In particular, the proposed method has the similar feedback-control framework with the work in [22] (see Fig. 1). However, there are two main differences comparing with [22].

First, there are two different methods between [22] and the proposed method in the deconvolution process. Shan et al. [22] take account of a density distribution prior. However, the proposed method utilizes FTVd method in the deconvolution process. Since the main step of FTVd method is fast Fourier transforms (FFTs) for each iteration, we can control the iterative number to get the faster speed and more accurate results than the decovolution method in [22].

Second, due to the inaccuracy of blur kernel in the deconvolution process, we employ a strategy of iterative kernel estimation to get more accurate kernel, aiming to make the resulted high-resolution image better. In contrast, Shan et al. [22] only utilize a fixed Gaussian kernel. However, the fixed kernel cannot get better results obviously.

## 4 Numerical experiments

In this section, we test the proposed method on two kinds of images. One is the low-resolution images without ground truth, and the other is the low-resolution images acquired by downsampling the ground truth images. All experiments are conducted in MATLAB(R2010a) on a laptop of 3.47 GB RAM and Intel(R) Core(TM) i3-2130M CPU: @3.40 GHz.

We make comparisons between the proposed method and some state-of-the-art image super-resolution methods, including bicubic interpolation, a fast upsampling method (“08’TOG” [22] ^{1}), a two-scale method (“14’TIP” [9] ^{2}) and two state-of-the-art interpolation methods (“11’IPOL” [7] ^{3} and “11’SIAM” [8] ^{4}).

For grayscale image, we apply the proposed algorithm directly. For colored image, we first transform the low-resolution image to YUV color space and then only conduct our algorithm on the Y channel. Images on the U and V channels are upsampled by bicubic method. After acquiring final upsampling image, we transform them from YUV space to RGB space for visual comparisons.

**Parameter selection:**There are four parameters in our algorithm: the size and the deviation of blur kernel, the regularization parameter

*α*, and the iteration number

*τ*. In fact, it is really difficult to choose appropriate parameters for many algorithms. Empirical adjustment is one of favorite ways for determining parameters. Thus, we select parameters empirically in our experiments. For the size and the deviation of blur kernel, the regularization parameter

*α*, we fix them according to different kinds of test images and different values of the upsampling factor (see details at Table 1). In particular, we use the Gaussian kernel similar to that described in “08’TOG” [22]. In addition, we estimate the errors on many images with different iteration numbers, and find that two or three iterations can get the best results. In particular, we set the iteration number

*τ*as 2 in all experiments to reduce the computation time.

Parameter selection in terms of blur kernel and regularization parameter (case 1 is for the low-resolution image without ground truth; case 2 is for the low-resolution images acquired by downsampling the ground truth images)

Upscale factor | Size | Deviation |
| |
---|---|---|---|---|

Case 1 | 2 | 5×5 | 1.25 | 10 |

3 | 7×7 | 1.85 | 10 | |

4 | 9×9 | 2.5 | 10 | |

Case 2 | 2 | 3×3 | 1.5 | 10 |

3 | 5×5 | 1.8 | 10 | |

4 | 7×7 | 2.3 | 10 |

### 4.1 Results on the low-resolution images without ground truth

Times of test images (unit: seconds)

### 4.2 Results on the low-resolution images acquired by downsampling the ground truth images

In this section, the low-resolution images are acquired by downsampling the ground truth images. We provide quantitative comparisons including root-mean-square error (RMSE), peak signal-noise ratio (PSNR), and structure similarity (SSIM) [41].

Quantitative comparison in terms of RMSE, PSNR and SSIM

Example(factor) | Index | Bicubic | 08’TOG [22] | 11’IPOL [7] | 11’SIAM [8] | 14’TIP [9] | Ours |
---|---|---|---|---|---|---|---|

House(2) | RMSE | 6.4742 | 6.0985 | 5.4645 | 5.3947 | 7.4008 | 5.1384 |

PSNR | 31.9071 | 32.4263 | 33.3798 | 33.4915 | 30.7452 | 33.9143 | |

SSIM | 0.8831 | 0.8858 | 0.8961 | 0.8969 | 0.8761 | 0.8958 | |

House(3) | RMSE | 8.9995 | 7.2818 | 7.8816 | 7.8106 | 9.1071 | 7.1034 |

PSNR | 29.0465 | 30.886 | 30.1985 | 30.2771 | 28.9432 | 31.1015 | |

SSIM | 0.847 | 0.8655 | 0.861 | 0.8615 | 0.8554 | 0.872 | |

House(4) | RMSE | 11.018 | 9.1549 | 9.8511 | 9.7658 | 11.7308 | 9.0703 |

PSNR | 27.2887 | 28.8977 | 28.2611 | 28.3366 | 26.7443 | 28.9784 | |

SSIM | 0.8169 | 0.8418 | 0.8302 | 0.8311 | 0.8278 | 0.8444 | |

Race(2) | RMSE | 11.18 | 11.2381 | 10.3093 | 10.1244 | 12.5453 | 10.8124 |

PSNR | 27.1619 | 27.1169 | 27.8662 | 28.0234 | 26.1612 | 27.4524 | |

SSIM | 0.6912 | 0.6911 | 0.7353 | 0.7386 | 0.6536 | 0.692 | |

Race(3) | RMSE | 13.9661 | 13.0151 | 13.0624 | 12.9765 | 14.1891 | 13.1136 |

PSNR | 25.2293 | 25.8418 | 25.8103 | 25.8676 | 25.0917 | 25.7764 | |

SSIM | 0.6042 | 0.6078 | 0.6407 | 0.6429 | 0.5955 | 0.6171 | |

Race(4) | RMSE | 15.7714 | 14.6136 | 14.8819 | 14.8098 | 15.7557 | 14.6909 |

PSNR | 24.1734 | 24.8357 | 24.6776 | 24.7198 | 24.182 | 24.7898 | |

SSIM | 0.5524 | 0.562 | 0.5804 | 0.5826 | 0.5575 | 0.5761 |

## 5 Conclusions

In this paper, we presented a simple and effective single image super-resolution method. Our method was motivated by a fast image upsampling method, but we differently studied the estimated error of the blur kernel in the proposed method. We believed that the point spread function is not known exactly in the process of image super-resolution. Through applying a fast total variation deconvolution (FTVd) strategy, we took an iterative updating strategy to update the blur kernel and the high-resolution image. In particular, the proposed method could be applied to any upscaling factors without any extra datasets. In addition, we analyzed the parameter selection and computation time. Extensive experiments were provided to illustrate the effectiveness of the proposed method.

## 6 Endnotes

## Declarations

### Acknowledgements

This research is supported by 973 Program (2013CB329404), NSFC (61370147, 61170311), the Fundamental Research Funds for the Central Universities (ZYGX2013Z005).

**Open Access** This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- S Baker, T Kanade, Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell.
**24:**, 1167–1183 (2002).View ArticleGoogle Scholar - S Farsiu, MD Robinson, M Elad, P Milanfar, Fast and robust multiframe super resolution. IEEE Trans. Image Process.
**13:**, 1327–1344 (2004).View ArticleGoogle Scholar - RY Tsai, TS Huang, Multiframe image restoration and registration. IEEE Trans. Image Process.
**1:**, 317–339 (1984).Google Scholar - R Keys, Cubic convolution interpolation for digital image processing. IEEE Trans. Speech Signal Process.
**29:**, 1153–1160 (1981).MathSciNetView ArticleMATHGoogle Scholar - LJ Deng, W Guo, TZ Huang, Single image super-resolution by approximated heaviside functions. Inform. Sci.
**348:**, 107–123 (2016).MathSciNetView ArticleGoogle Scholar - D Zhang, X Wu, An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process.
**15:**, 2226–2238 (2006).View ArticleGoogle Scholar - P Getreuer, Image interpolation with contour stencils. Image Processing On Line(2011), http://www.ipol.im/.
- P Getreuer, Contour stencils: total variation along curves for adaptive image interpolation. SIAM J. Imaging Sci.
**4:**, 954–979 (2011).MathSciNetView ArticleMATHGoogle Scholar - L Wang, H Wu, C Pan, Fast image upsampling via the displacement field. IEEE Trans. Image Process.
**23:**, 5123–5135 (2014).MathSciNetView ArticleGoogle Scholar - X Li, MT Orchard, New edge-directed interpolation. IEEE Trans. Image Process.
**10:**, 1521–1527 (2001).View ArticleGoogle Scholar - LJ Deng, W Guo, TZ Huang, Single image super-resolution via an iterative reproducing kernel hilbert space method. IEEE Trans. Circuits and Systems for Video Technology (2015). doi:10.1109/TCSVT.2015.2475895.
- S Borman, RL Stevenson, Super-resolution from image sequences-a review, Midwest Symposium on Circuits and Systems (MWSCAS). Notre Dame, Indiana, pp. 374–378 (1998).Google Scholar
- S Rhee, MG Kang, Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Eng.
**38:**, 1348–1356 (1999).View ArticleGoogle Scholar - AK Katsaggelos, KT Lay, NP Galatsanos, A general framework for frequency domain multi-channel signal processing. IEEE Trans. Image Process.
**2:**, 417–420 (1993).View ArticleGoogle Scholar - S Lertrattanapanich, NK Bose, High resolution image formation from low resolution frames using delaunay triangulation. IEEE Trans. Image Process.
**11:**, 1427–1441 (2002).MathSciNetView ArticleGoogle Scholar - H Stark, P Oskoui, High-resolution image recovery from image-plane arrays, using convex projections. J. Optical Soc. Am. A.
**6:**, 1715–1726 (1989).View ArticleGoogle Scholar - D Capel, A Zisserman, Super-resolution enhancement of text image sequences. Intern. Conf. Pattern Recognit.
**1:**, 600–605 (2000).View ArticleGoogle Scholar - R Fattal, Image upsampling via imposed edge statistics. ACM Trans. Graphics (TOG). 26(3) (2007). Article No.95.Google Scholar
- T Komatsu, T Igarashi, K Aizawa, T Saito, Very high resolution imaging scheme with multiple different-aperture cameras. Signal Process. Image Commun.
**5:**, 511–526 (1993).View ArticleGoogle Scholar - A Chambolle, T Pock, A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis.
**40:**, 120–145 (2011).MathSciNetView ArticleMATHGoogle Scholar - M Irani, S Peleg, Motion analysis for image enhancement: resolution, occlusion, and transparency. journal of visual communication and image representation. J. Visual Commun. Image Repres.
**4:**, 324–335 (1993).View ArticleGoogle Scholar - Q Shan, Z Li, J Jia, CK Tang, Fast image/video upsampling. ACM Trans. Graphics (TOG).
**27:**, 32–39 (2008).Google Scholar - P Chatterjee, S Mukherjee, S Chaudhuri, G Seetharaman, Application of papoulis-gerchberg method in image super-resolution and inpainting. Comput. J.
**52:**, 80–89 (2009).View ArticleGoogle Scholar - H Takeda, S Farsiu, P Milanfar, Kernel regression for image processing and reconstruction. IEEE Trans. Image Process.
**16:**, 349–366 (2007).MathSciNetView ArticleGoogle Scholar - X Qinlan, C Hong, C Huimin, Improved example-based single-image super-resolution. Intern. Congr. Image Signal Process. (CISP).
**3:**, 1204–1207 (2010).View ArticleGoogle Scholar - C Kim, K Choi, K Hwang, JB Ra, Learning-based superresolution using a multi-resolution wavelet approach, International Workshop Advance. Image Tech (IWAIT), Gangwon-Do, South Korea (2009).Google Scholar
- C Dong, CC Loy, K He, X Tang, Learning a deep convolutional network for image super-resolution. European Conference on Computer Vision (ECCV), Zurich, Switzerland, pp. 184–199 (2014).Google Scholar
- J Yang, J Wright, TS Huang, Y Ma, Image super-resolution via sparse representation. IEEE Trans. Image Process.
**19:**, 2861–2873 (2010).MathSciNetView ArticleGoogle Scholar - K Jia, X Wang, X Tang, Image transformation based on learning dictionaries across image spaces. IEEE Trans. Pattern Anal. Mach. Intell.
**35:**, 367–380 (2013).View ArticleGoogle Scholar - J Yang, Z Wang, Z Lin, S Cohen, T Huang, Coupled dictionary training for image super-resolution. IEEE Trans. Image Process.
**21:**, 3467–3478 (2012).MathSciNetView ArticleGoogle Scholar - WT Freeman, TR Jones, EC Pasztor, Example-based super-resolution. IEEE Comput. Graph. Appl.
**22:**, 56–65 (2002).View ArticleGoogle Scholar - WT Freeman, EC Pasztor, OT Carmichael, Learning low-level vision. Intern. J. Comput. Vis.
**40:**, 25–47 (2000).View ArticleMATHGoogle Scholar - N Efrat, D Glasner, A Apartsin, B Nadler, A Levin, Accurate blur models vs. image priors in single image super-resolution. IEEE Intern. Conf. Computer Vision (ICCV), Sydney, Australia (2013).Google Scholar
- Y Wang, JC Yang, W Yin, Y Zhang, A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci.
**1:**, 248–272 (2008).MathSciNetView ArticleMATHGoogle Scholar - XL Zhao, F Wang, TZ Huang, MK Ng, RJ Plemmons, Deblurring and sparse unmixing for hyperspectral images. IEEE Trans. Geosci. Remote Sensing.
**51:**, 4045–4058 (2013).View ArticleGoogle Scholar - S Wang, TZ Huang, J Liu, XG Lv, An alternating iterative algorithm for image deblurring and denoising problems. Commun. Nonlin. Sci. Numer. Simul.
**19:**, 617–626 (2014).MathSciNetView ArticleGoogle Scholar - G Liu, TZ Huang, J Liu, High-order tv l1-based images restoration and spatially adapted regularization parameter selection. Comput. Math. App.
**67:**, 2015–2026 (2014).View ArticleGoogle Scholar - XL Zhao, F Wang, MK Ng, A new convex optimization model for multiplicative noise and blur removal. SIAM J. Imaging Sci.
**7:**, 456–475 (2014).MathSciNetView ArticleMATHGoogle Scholar - LJ Deng, H Guo, TZ Huang, A fast image recovery algorithm based on splitting deblurring and denoising. J. Comput. Appl. Math.
**287:**, 88–97 (2015).MathSciNetView ArticleMATHGoogle Scholar - XL Zhao, W Wang, TY Zeng, TZ Huang, MK Ng, Total variation structured total least squares method for image restoration. SIAM J. Sci. Comput.
**35:**, 1304–1320 (2013).MathSciNetView ArticleMATHGoogle Scholar - Z Wang, AC Bovik, HR Sheikh, EP Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process.
**13:**, 600–612 (2004).View ArticleGoogle Scholar