A combined total variation and bilateral filter approach for image robust super resolution
 Amine Laghrib^{1}Email author,
 Abdelilah Hakim^{1} and
 Said Raghay^{1}
https://doi.org/10.1186/s1364001500754
© Laghrib et al. 2015
Received: 14 February 2015
Accepted: 9 June 2015
Published: 26 June 2015
Abstract
In this paper, we consider the image superresolution (SR) reconstitution problem. The main goal consists of obtaining a highresolution (HR) image from a set of lowresolution (LR) ones. For that, we propose a novel approach based on a regularized criterion. The criterion is composed of the classical generalized total variation (TV) but adding a bilateral filter (BTV) regularizer. The main goal of our approach consists of the derivation and the use of an efficient combined deblurring and denoising stage that is applied on the highresolution image. We demonstrate the existence of minimizers of the combined variational problem in the bounded variation space, and we propose a minimization algorithm. The numerical results obtained by our approach are compared with the classical robust superresolution (RSR) algorithm and the SR with TV regularization. They confirm that the proposed combined approach allows to overcome efficiently the blurring effect while removing the noise.
Keywords
Super resolution Bilateral filter Bounded variation space Total variation Function relaxed1 Introduction
The problem of the reconstruction of a superresolution image from lowresolution ones is required in numerous applications such as video surveillance [1], medical diagnostics [2] and image satellite [3].
A socalled fast robust superresolution procedure was proposed in [4]. In this approach, Farsiu et al. proposed a twostage approach. In the first stage, a highresolution image is built, but having the problem of being blurred. Then, in the second stage, a deblurring and denoising procedure is considered, see [4, 5]. Our paper will focus on this second stage in the context of super resolution. The main goal consists of increasing the robustness of the superresolution (SR) technique in [4] with respect to the blurring effect and to the noise.
In most cases, the problem of image deblurring or denoising is an illposed one. It is the main reason why the problem is considered as an optimization one, but considering a regularized criterion. Some of the widely used regularization functions are Tikhonovtype regularizer [6, 7] and total variationtype regularizer [4, 8, 9]. In the following, we will consider a total variation (TV) regularization framework, but adding a bilateral filtering part [4, 10]. The main point of this combination mainly consists of preserving the essential features of the image such as boundaries and corners that are degraded, using other approaches.
The outline of the paper is as follows. In Section 2, we present the general superresolution problem. Then, in Section 3, we present the proposed regularized criterion after pointing out the different regularization used in the literature. Hence, we introduce the variational problem and we prove the existence of a minimizing solution of the relaxed functional using standard techniques from calculus of variations. In Section 4, we derive the proposed algorithm, and in Section 5, we present some experimental results; in addition, we compare our approach with some existing ones in the literature. We finally end the paper by a conclusion.
2 Problem formulation
The observed images of a real scene usually are in low resolution. This is due to some degradation operators. Moreover, in practice, the acquired images are decimated, corrupted by noise and suffered from blurring [11–13]. We assume that all lowresolution images are taken under the same environmental conditions using the same sensor.
where H is the blurring operator of size [r ^{2} N ^{2}×r ^{2} N ^{2}], D represents the decimation matrix of size [N ^{2}×r ^{2} N ^{2}], F _{ k } is a geometric warp matrix of size [r ^{2} N ^{2}×r ^{2} N ^{2}], representing a nonparametric transformation that differs in all frames, and e _{ k } is a vector of size [N ^{2}×1] that represents the additive noise for each image.
 1.
Computing the warp matrix F _{ k } for each image.
 2.
Fusing the LR images Y _{ k } into a blurred HR version B=H X.
 3.
Finding the estimation of the HR image X from the blurring and noised one B.
We will not detail the first and second steps in the following sections; for more details, see [5, 14]. We will focus on the last step which is a deconvolution and denoising step.
3 Deconvolution and denoising step
the norm of the Lebesgue space L^{1}(Ω): \(\Vert HX\widehat {B} \Vert _{1}\), is used since it is very robust against outliers [4]. p(X) denotes the prior knowledge on the HR image, described by the prior Gibbs function (PGF). We present in the following subsection the related work to the choice of the PGF function.
3.1 Related work
where Γ is a highpass operator such as Laplacian.
where f is strictly convex and nondecreasing function from \(\mathbb {R}^{+}\) to \(\mathbb {R}^{+}\) such as f(0)=0 and \(\lim \limits _{x \to +\infty }f(x)=+\infty \). The choice of this PGF function was typically in the denoising and deblurring process in many restoration problems [18] since it preserves edges in the reconstruction, but sometimes causes some artificial edges in the smooth surfaces.
The operators \({S^{i}_{x}}\) and \({S^{j}_{y}}\) shift X by i and j pixels in horizontal and vertical directions, respectively, presenting several scales of derivatives. The scalar weight α (0<α<1) is applied to give a spatially decaying effect to the summation of the regularization terms. p is the spatial window size and i+j>0.
where the parameter c is the threshold and ρ(x,c) the potential function that penalize the gradient.
3.2 The proposed regularization
Since this space is nonreflexive, we cannot say anything about a bounded minimizing sequence in W ^{1,1}(Ω). To overcome the illposedness of this problem, we use the procedure of relaxation. A typical choice of the space that guarantees the compactness results is the space of functions of bounded variation B V(Ω) [18].
3.2.1 3.2.1 The Proprieties of B V(Ω) space
We summarize firstly some of the properties of the space B V(Ω) that we will use in the following theorems. We suppose in the following that Ω is bounded and has a Lipschitz boundary.
( P _{ 1 } ) Lower semicontinuity (l.s.c) in B V ( Ω )
( P _{ 2 } ) The weak* topology in B V ( Ω )
\(\mathcal {C}^{1}_{0}(\Omega)^{N}\) is the space of continuously differentiable functions with compact support in Ω.

The space B V(Ω) is continuously embedded in L^{2}(Ω) (N = 2 the dimension of the space).

Every uniformly bounded sequence (X _{ j }) in B V(Ω) is relatively compact in L^{ p }(Ω) for \(1 \leq p < \frac {N}{N1}, N \geq 1\). Moreover, there exists a subsequence (X _{ jk }) and X∈B V(Ω) such as \(X_{\textit {jk}}\overset {}{\underset {BV\omega *} {\rightharpoonup }} X \).
For the reason that every bounded sequence in W ^{1,1}(Ω) is also bounded in B V(Ω), we use the classical characteristics of the B V−ω∗ topology to deduce the existence of a subsequence that converges B V−ω∗. Let us define the relaxed function of the problem (9).
Theorem 3.1.
\(\mathcal {H}\)is the Hansdorff measure and C _{ x } the Cantor part.
Let us prove now the existence of the problem (9).
Theorem 3.2.
admits a solution X∈B V(Ω).
i.e. X is a minimum of \(\overline {F}\).
For the uniqueness, we cannot say anything since the L^{1} norm is not strictly convex. However, if we replace the norm ∥H X _{ k }−B∥_{1} by \(\Vert {HX}_{k}B {\Vert _{2}^{2}}\), we can check easily the uniqueness of the solution.
4 Proposed algorithm
5 Numerical results
The PSNR table
Image  Method  σ=10  σ=15  σ=20 
Lena  SR with TV reg.  27.2222  26.868  26.426 
RSR  28.07  27.78  27.589  
Proposed approach  29.0844  28.7012  28.5562  
Barbara  SR with TV reg.  26.0826  25.658  25.4893 
RSR  25.6836  25.1263  25.0369  
Proposed approach  26.6194  26.2022  26.0014  
Bird  SR with TV reg.  33.0900  32.6237  32.254 
RSR  33.1751  32.8233  32.5865  
Proposed approach  34.8474  34.5266  34.33  
Lake  SR with TV reg.  30.9070  30.25  29.922 
RSR  30.6298  30.2522  30.0866  
Proposed approach  31.0437  30.86  30.636  
Baboon  SR with TV reg.  26.0667  25.789  25.3244 
RSR  25.7250  25.388  25.263  
Proposed approach  27.4975  27.1626  26.92  
Peppers  SR with TV reg.  29.9331  29.544  29.1668 
RSR  30.9049  30.5012  30.278  
Proposed approach  30.9569  30.68  30.4622 
6 Conclusions
We propose a new combination of TV and BTV in the space of bounded variation applied in the deblurring step of the robust superresolution problem. We prove the existence of minimizers using a relaxation technique. Finally, we perform the choice of our model using the PSNR criteria in Section 5.
Declarations
Acknowledgements
We are grateful to the anonymous referee for the corrections and useful suggestions that have improved this article.
Authors’ Affiliations
References
 Q Luong, Advanced image and video resolution enhancement techniques. PhD thesis, Faculty of Engineering Ghent University, (2009).Google Scholar
 D Hill, J Hajnal, D Hawkes (eds.), Medical Image Registration (CRC, 2001).Google Scholar
 H Zhang, Z Yang, Li Zhang, H Shen, Superresolution reconstruction for multiangle remote sensing images considering resolution differences. Remote Sens. 6, 637–657 (2014).View ArticleGoogle Scholar
 S Farsiu, M Dirk, M Elad, P Milanfar, Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13, 1327–1344 (2004).View ArticleGoogle Scholar
 A Laghrib, A Hakim, S Raghay, M EL Rhabi, Robust super resolution of images with nonparametric deformations using an elastic registration. Appl. Math. Sci. 8, 8897–8907 (2014).Google Scholar
 E Lee, M Kang, Regularized adaptive highresolution image reconstruction considering inaccurate subpixel registration. IEEE Trans. Image Process. 12, 806–813 (2003).Google Scholar
 V Patanavijit, S Jitapunkul (eds.), A Robust Iterative Multiframe Superresolution Reconstruction Using a Huber Bayesian Approach with Huber–Tikhonov Regularization (International Symposium on Intelligent Signal Processing and Communications, Yonago, Japan, 2006).Google Scholar
 M Ng, H Shen, E Lam, L Zhang, A total variation regularization based superresolution reconstruction algorithm for digital video. EURASIP J. Adv. Signal Process, 1–16 (2007). Article ID 74585.Google Scholar
 L Rudin, S Osher, E Fatemi, Nonlinear total variation based noise removal algorithms. Physica D. 60, 259–268 (1992).MATHView ArticleGoogle Scholar
 X Zenga, L Yangi, A robust multiframe superresolution algorithm based on halfquadratic estimation with modified BTV regularization. Digital Signal Process. 23, 98–109 (2013).View ArticleGoogle Scholar
 P Milanfar, SuperResolution Imaging (Digital Imaging and Computer Vision) (Taylor and Francis/CRC Press, 2010).Google Scholar
 RY Tsai, TS Huang, Multiframe image restoration and registration. Advances in Computer Vision and Image Processing, vol. 1, chap. 7 (JAI Press, Greenwich, Conn, USA, 1984).Google Scholar
 S Park, M Park, M Kang, Superresolution image reconstruction : a technical overview. 20(3), 21–36 (2003).Google Scholar
 A Laghrib, A Hakim, S Raghay, ME Rhabi, A robust multiframe super resolution based on curvature registration and second order variational regularization. Int. J. Tomography Simul. 28, 63–71 (2015).Google Scholar
 MK Park, MG Kang, Regularized highresolution reconstruction considering inaccurate motion information. Opt. Eng. 46, 117004 (2007).View ArticleGoogle Scholar
 V Patanavijit, S Jitapunkul, A lorentzian stochastic estimation for a robust iterative multiframe superresolution reconstruction with Lorentzian–Tikhonov regularization. EURASIP J. Adv. Signal Process, 1–21 (2007).Google Scholar
 S Osher, M Burger, D Goldfarb, J Xu, W Yin, An iterative regularization method for total variationbased image restoration. Multiscale Model. Simul. 4, 460–489 (2005).MATHMathSciNetView ArticleGoogle Scholar
 G Aubert, P Kornprobst, Mathematical Problems in Image Processing Partial Differential Equations and the Calculus of Variations. Second Edition (Springer, New York, 2006).Google Scholar
 F Demengel, G Demengel, Espaces Fonctionnels. Utilisation dans la Résolution des équations aux Dérivées Partielles, Savoirs actuels (EDP Sciences, 2007).Google Scholar
 L Ambrosio, A Compactness Theorem for a New Class of Functions of Bounded Variation (Boll. Un. Mat. Ital., 1989).Google Scholar
 F Demengel, R Temam, Convex functions of a measure and applications. Indiana Univ. Math. J. 33, 673–709 (1984).MATHMathSciNetView ArticleGoogle Scholar
 L Vese, Problèmes variationnels et edp pour l‘analyse d’images et l‘évolution de courbes. PhD thesis, Université de Nice SophiaAntipolis, Nov. (1996).Google Scholar
 H Brezis, Functional Analysis, Sobolev Spaces and Partial Differential Equations (Springer, New York, 2011).Google Scholar
 Y Zhang, J Xu, F Dai, J Zhang, Q Dai, F Wu, Efficient parallel framework for HEVC motion estimation on manycore processors. IEEE Trans. Circ. Syst. Video Technol. 24 (2014).Google Scholar
Copyright
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.