A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer

Abstract

In this work, the authors have proposed a multi-frame super-resolution method that is based on the diffusion-driven regularization functional. The new regularizer contains a variable exponent that adaptively regulates its diffusion mechanism depending upon the local image features. In smooth regions, the method favors linear isotropic diffusion, which removes noise more effectively and avoids unwanted artifacts (blocking and staircasing). Near edges and contours, diffusion adaptively and significantly diminishes, and since noise is hardly visible in these regions, an image becomes sharper and resolute—with noise being largely reduced in flat regions. Empirical results from both simulated and real experiments demonstrate that our method outperforms some of the state-of-the-art classical methods based on the total variation framework.

1 Introduction

1.1 Overview of the super-resolution methods

Super-resolution image reconstruction refers to a collection of various image-processing tools used to reconstruct high-resolution images from their corresponding degraded versions ([1–5], and references therein). The term resolution, as used in this work, is the number of pixels per unit area. Therefore, a high-resolution image with both appreciable subjective and objective qualities has a larger pixel count compared with a low-resolution image.

Humans are naturally inclined to vouch for high-quality images. Perhaps the reason for this inclination is the fact that such images contain more information and, hence, are easier to comprehend and interpret. In machine learning and computer vision tasks, detailed images are useful to accurately and robustly highlight and extract critical features, such as edges and contours. These typical examples make super-resolution a vital technology.

Super-resolution methods attempt to address the hardware limitations for improving image qualities. The advantage of these methods is that they discourage hardware modifications to achieve high-resolution images and, thus, are cheap and promote portability. The hardware approach, on the contrary, prompts for changes in the sensor, chip size, and device’s internal circuitry to meet the same goal of the super-resolution process. These hardware changes come at the following costs: (1) the current technology disallows further reduction of the sensor’s pixel size (which increases pixel count and, therefore, improves resolution) as the process may introduce unwanted shot noise that degrades images; (2) increasing the chip size, which improves the spatial resolution, raises capacitance that slows down the charge transfer rate; and (3) additional weight due to hardware change may limit particular applications, such as remote sensing or unmanned aerial surveillance.

The methods for solving the super-resolution problem can be put into two groups, namely multi-frame and single-frame. The former group uses a single image to generate its corresponding high-resolution version [6–9]. Despite the extensive review, single-frame-based super-resolution approaches—interpolation (nearest neighbor, bilinear, and bicubic) and example-based [10–13]—tend to generate unpleasing results due to insufficient amount of input images. For example, interpolation methods are known for producing blurry and aliased images. Besides, example-based approaches are computationally inefficient and may, therefore, be unsuitable for real-time processing. A thorough discussion on single-frame super-resolution methods is beyond the scope of our work. However, for a comprehensive review of this category of methods, we refer interested readers to the works in[7, 8, 14–20]

A multi-frame super-resolution approach—which this paper is based upon—attempts to generate a high-resolution image by fusing some pieces of information from multiple degraded images of the same scene [21–24]. The fundamental premise of this approach is that pairs of similar images are likely to differ by rotations, translations, and/or affine transformations due to (1) shaking of the camera, (2) different capturing and exposure times, and (3) relative motion between camera and scene. We can exploit this additional information to produce a more resolute scene by computing the values of these attributes and using them to align the frames onto a common high-resolution grid. The rest of this work discusses the multi-frame super-resolution methods.

The super-resolution problem is highly ill-posed due to insufficient number of the observed low-resolution frames. A classical approach to address this problem is called regularization. Authors have, therefore, proposed different types of priors.

A classical and, perhaps, popular prior is based on the Tikhonov model [25, 26]. This prior introduces into the restoration problem a smoothness constraint that removes extraneous noise from an image. The weakness of the Tikhonov prior is that it tends to destroy edges—an effect that degrades images. Therefore, the prior has captured the interest of many researchers to develop models that simultaneously suppress noise and preserve critical image features: Huber Markov random field (Huber-RMF) [27, 28], edge-adaptive RMF [29, 30], sparse directional [31, 32], and total variation (TV) [33–35].

Of the aforementioned priors, TV has attracted more attention of researchers as it generates results with pleasing objective and subjective qualities. The major weaknesses of the TV model are blocking and staircasing effects, false-edge generation near edges, and non-differentiability property at zero—a situation that makes the numerical implementation rather challenging. The TV model was initially applied in image denoising [36]. Later, the model was adapted to other applications: super-resolution [33], MRI medical image reconstruction [37], inpainting [38], and deblurring [39]. In this work, we explore some classical TV-based approaches to address the super-resolution problem.

In 2008, Marquina et al. proposed a convolutional model for a super-resolution problem based on the constrained TV framework [33]. In their work, the authors introduced the Bregman algorithm as an iterative refinement step to enhance the spatial resolution. The results demonstrate that the method generates detailed and sharper images, but blocking and staircasing artifacts are still evident.

In [35], Farsiu and colleagues proposed a bilateral total variation (BTV) prior, which is based on the L ¹ norm minimization and the bilateral filter regularization functional, for a multi-frame super-resolution problem. Their method is computationally inexpensive and robust against errors caused by motion and blur estimations and generates images that are convincingly sharper. However, bilateral filters that the method derives its strengths from are known to introduce artifacts like staircasing and gradient reversal. Additionally, the BTV inadequately addresses the partial smoothness of an image [40]. Besides, the numerical implementation of the L ¹ norm component is challenging as it masks the super-resolution data term.

In [34], Ng et al. applied the TV prior to address the following issues in the super-resolution video reconstruction: noise, blurring, missing regions, compression artifacts, and motion estimation errors. The authors demonstrated the efficacy of their method in several cases of motions and degradations and provided the experimental results that outperform some other classical super-resolution methods.

Ren et al. proposed a super-resolution method, which is based on the fractional order TV regularization, with a focus to handle fine details, such as textures, in the image [41]. Results show that their approach addresses to some extent the weaknesses of the traditional TV.

In [21], Li et al. attempted to address the drawbacks of the global TV by proposing two regularizing functionals, namely locally adaptive TV and consistency of gradients, to ensure that edges are sharper and flat regions are smoother. The method heavily depends on the gradient details of an image, a feature that may produce pseudo-edges in noisy homogeneous regions. Note that both noise and edges are image features with high-gradient (or high-intensity) values. As Li’s method is gradient-dependent, it may equally treat both noise and edges, and this may generate unwanted artifacts.

Yuan et al. proposed a spatially weighted TV model for the multi-frame super-resolution problem [42]. Their model incorporates a spatial information indicator (difference curvature) that locally identifies the spatial properties of the individual pixels, thus providing the necessary level of regularization. The authors employed the majorization-minimization algorithm to optimize their formulation. Results show that the Yuan et al. method overcomes some challenges of the original TV model (discourages piecewise constant solutions and is less sensitive to regularization parameters) and outperforms the Li et al. method. But under severe noise conditions, the Yuan et al. approach fails as it is pixel-unit-based [40].

Recently, Weili et al. proposed an adaptive TV-driven super-resolution method that provides convincing results compared with those generated by the standard TV model [40]. The authors incorporated into the classical super-resolution formulation a feature-sensitive prior that approximates the L ¹ norm near edges, and this helps to efficiently highlight these critical features. In flat regions, the prior approximates the L ² norm, thus providing noise removal.

Inspired by the weaknesses of the standard TV and its variants, this work proposes an alternative framework based on nonlinear diffusion processes—which have yielded promising results in image-denoising applications [36, 43–47]—to address the super-resolution ill-posedness. Our diffusion-driven prior includes an adaptive kernel that sensitively and dynamically updates its value in accordance with the scanned local image features. That is, the kernel is linear isotropic in flat regions and nonlinear anisotropic near edges. Being locally adaptive, the model generates more resolute scenes and avoids blocking artifacts inherent in the conventional TV.

1.2 A classical multi-frame image degradation model

Imaging devices, such as cameras, endure unavoidable hardware limitations and influence of external noise. Therefore, an image signal that passes through the imaging system usually undergoes degradations before reaching its destination (Fig. 1).

Fig. 1

Image degradation model for a multi-frame super-resolution problem

Let us consider that u is the unknown high-resolution image. In the image degradation model (Fig. 1), u is first acted upon by the warping operator, W _k , which rotates and translates it. Then, the degraded version of u proceeds to the next stage, where it is blurred by the blurring operator, B _k . In the next stage, the dimension (width × height) of the degraded (warped and blurred) u is decreased by the decimation operator, D _k . Finally, noise (assumed to be additive white Gaussian), η _k , is added as a further image degradation agent. Consequently, a sequence of low-resolution images, {y _k }, where k=1,2,3,…,M is generated. As the model is multi-frame-based, the degradation process incorporates a sequence of composite operators— W ₁ B ₁ D ₁,…,W _k B _k D _k ,…,W _M B _M D _M —which are applied to u to produce y ₁,…,y _k ,…,y _M , respectively.

Therefore, we intuitively represent the multi-frame image degradation model by the equation

$$ y_{k}=W_{k}B_{k}D_{k}u+\eta_{k} $$

((1))

Treating η _k as energy, our objective is to minimize it using the L ² norm that is known for its ability to suppress noise. Therefore, the minimization problem becomes

$$ \min_{u}\left\{\frac{1}{2M}\sum_{k=1}^{M}\|y_{k}-W_{k}B_{k}D_{k}u\|_{L^{2}(\Omega)}^{2}\right\}, $$

((2))

where Ω is the supporting domain of u. Applying the Euler-Lagrange optimization approach, and embedding the resulting solution into a dynamical system, we get

$$ \frac{\partial u}{\partial t}=\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}\left(W_{k}B_{k}D_{k}u-y_{k}\right). $$

((3))

2 Multi-frame super-resolution process

2.1 Background

The fundamental aim of the super-resolution process is to improve the spatial resolution of an image while suppressing unwanted artifacts. A well-known approach to achieve this goal is to incorporate a regularizing potential into the super-resolution energy functional. Therefore, we can reformulate (2) as a variational problem

$$ {\fontsize{8.8pt}{9.6pt}\selectfont{\begin{aligned} {}\min_{u\in BV\cap L^{2}(\Omega)}&\!\left\{\!\frac{1}{2M}\int_{\Omega}~\sum_{k=1}^{M}(y_{k}\,-\,W_{k}B_{k}D_{k}u)^{2}dx\,+\,\int_{\Omega}\psi(|\nabla u|)|\nabla\! u|dx\right.\\ &+\left.\frac{\lambda}{2}\int_{\Omega}(u-f)^{2}dx\vphantom{\frac{1}{2M}\int_{\Omega}\sum_{k=1}^{M}(y_{k}-W_{k}B_{k}D_{k}u)^{2}dx}\right\}, \end{aligned}}} $$

((4))

where the terms from left are defined as follows: super-resolution or data, regularization potential, and fidelity, respectively. From (4), the coefficient function, ψ(s), simultaneously detects edges and penalizes the norm of the image gradient, and the fidelity term, which contains λ as the regularizing parameter, establishes a trade-off between the evolving image, u, and the initial guess, f.

The parameterized (time-dependent) Euler-Lagrange form of (4) is

$$ \begin{aligned} \frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})\\&+\text{div}\left(\left(\frac{\psi(|\nabla u|)}{|\nabla u|}+\psi^{\prime}(|\nabla u|)\right)\nabla u\right)\\&-\lambda(u-f). \end{aligned} $$

((5))

In [36], Rudin et al. proposed ψ(s)=1, and plugging this value into (5) yields

$$\begin{array}{*{20}l} {}\frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})+\text{div}\left(\frac{1}{|\nabla u|}\nabla u\right)\\&-\lambda(u-f), \end{array} $$

((6))

which is the multi-frame super-resolution model based on the Rudin-Osher-Fatemi (ROF) regularizing functional. Although the ROF model suppresses noise and effectively recovers edges, it has some limitations, as noted by Ogada and colleagues [43]. Firstly, the formulation favors piecewise-constant solutions that result into staircasing effects or even generation of false edges. Secondly, the ROF model tends to reduce contrast in homogeneous or noise-free image regions. Thirdly, the TV diffusion model—despite its anisotropic property—produces a process that only diffuses in a direction that is orthogonal to the gradients of the image contours. This has a consequence of producing blockiness in the results. And lastly, the ROF evolution system contains a $\frac {1}{|\nabla u|}$ component that runs into a spike when |∇u|=0 in flat regions. To solve this challenge, the numerical implementations of the model usually incorporate a lifting parameter 0<ε<1, where ε is made too small. That is, the diffusion coefficient in the divergence part of (6) is replaced by $\frac {1}{|\nabla u|+\epsilon }$. In our view, this modification limits the accuracy of the results and may even cause instabilities.

Motivated by the weaknesses of the ROF model, Ogada et al. proposed an alternative method for image denoising that focuses on selecting an appropriate value of ψ(s). Their approach provides a criteria for deciding the nature of ψ(s) in terms of linearity, sub-linearity, and super-linearity. This ensures that the resulting regularizing functional is strictly convex and grows linearly. In image-denoising problems, linear growth and convex functionals are known to generate appealing results. Additionally, the authors’ diffusion equation contains the denominator that never collapses to zero even in smooth regions (where |∇u|=0), thus avoiding the singularity risk like that encountered in the TV problem.

Therefore, Ogada et al. selected a sub-linear growth functional coefficient

$$ \psi(|\nabla u|)=\frac{|\nabla u|}{1+|\nabla u|}. $$

((7))

Plugging ψ(s) from (7) into (5) gives

$$ {\fontsize{8.8pt}{9.6pt}\selectfont{\begin{aligned} {}\frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})+\text{div}\left(\frac{2+|\nabla u|}{(1+|\nabla u|)^{2}}\nabla u\right)\\&-\lambda(u-f). \end{aligned}}} $$

((8))

Decomposing (7) into $u_{_{\textit {TT}}}$ (tangential) and $u_{_{\textit {NN}}}$ (normal) components yields

$$\begin{array}{*{20}l} {}\frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u\,-\,y_{k})\,+\,\left(\!\frac{2+|\nabla u|}{(1+|\nabla u|)^{2}}\!\right)u_{_{TT}}\\&+\left(\frac{2}{(1+|\nabla u|)^{3}}\right)u_{_{NN}}-\lambda(u-f), \end{array} $$

((9))

which possesses the diffusion mechanisms in both tangential and normal directions to the isophote lines. Furthermore, Eq. 9 promotes varying degrees of diffusions depending upon the local image structures, particularly edges and contours.

Therefore, in order to reap the benefits of the model by Ogada et al. and, indeed, the edge-preserving capability and sensitivity of the model to the finer local image structures, we have exponentiated the denominator of the diffusivity of (8) by an adaptive term. Our goal was to make the smoothing functional adapt several models: isotropic diffusion, ROF, and Ogada et al., and to ensure global minimum energy that guarantees uniqueness in the results.

2.2 Proposed model

2.2.1 2.2.1 Model formulation

The smoothing functional proposed by Ogada et al. [43] produces superior results compared with the Perona-Malik (PM) [44], D- α-PM [45], and total variation [36]. The model has been used for denoising applications. In this work, we have modified the model by integrating into its diffusivity an edge-probing variable exponent that is robust against noise. Furthermore, the modified formulation was encapsulated into the classical multi-frame super-resolution model.

Now, we propose a non-variational problem

$$ {\fontsize{8.8pt}{9.6pt}\selectfont{\begin{aligned} {}0=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})\,+\,\text{div}\left(\!\frac{2+\frac{|\nabla u|}{\beta}}{\left(1+\frac{|\nabla u|}{\beta}\right)^{\alpha(x)}}\nabla u\right)\\&-\lambda(u-f), \end{aligned}}} $$

((10))

where β is a shape-defining parameter and

$$ \alpha(x)=2-\frac{1}{1+\kappa|\nabla (G_{\sigma}\ast f)|^{2}} $$

((11))

is an adaptive feature-dependent variable exponent with σ>0 and $G_{\sigma }(x)=\frac {1}{4\pi \sigma }\exp \left (\frac {-|x|^{2}}{4\sigma ^{2}}\right)$ (Fig. 2). The authors in [46] found that σ=0.50 and 0.0025<κ<0.025 produced promising restoration results, and indeed, their findings worked well for our case. Equation 10 is usually solved by embedding it into a dynamical system, which is then evolved until steady state conditions are attained. Therefore, parameterizing the equation in time yields

$$ {\fontsize{8.8pt}{9.6pt}\selectfont{\begin{aligned} {}\frac{\partial u}{\partial t}\!=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}\left(W_{k}B_{k}D_{k}u\,-\,y_{k}\right)\!+\text{div}\left(\!\frac{2+\frac{|\nabla u|}{\beta}}{\left(1+\frac{|\nabla u|}{\beta}\right)^{\alpha(x)}}\nabla u\!\right)\\&-\lambda(u-f) \text{~~for~~}(x,t)\in\Omega\times(0,T),\\ &u(x,0)=f,~~~x\in\Omega,\\ &\frac{\partial u}{\partial\vec{n}}=0,~~~(x,t)\partial\Omega\times(0,T), \end{aligned}}} $$

((12))

Fig. 2

Variation of the feature-dependent variable exponent, α(x), with respect to the image gradient

which can be implemented in the computer using the appropriate numerical schemes. In this paper, we used the four-point neighborhood explicit scheme to implement (12), as detailed in the later sections.

Figure 3 shows the block diagram that implements our model in (12). From the illustration, a sequence of M frames, y ₁,y ₂,…,y _k ,…,y _M , is first captured by an imaging device. Then, the values of the motion parameters (rotations and translations) between {y _k+1} _k=1,2,…,M and y ₁ (reference frame) are computed using the Keren et al. algorithm—which is chosen for its reliability, robustness, accuracy, and computational efficiency. Next, the frames are aligned using the computed motion values and projected onto the high-resolution grid. Next, the Zomet et al. method is used to robustly detect outliers in the data. The actual reconstruction is done in the following step, and we used the steepest descent approach for this purpose. Lastly, the proposed regularizing functional is incorporated to address the super-resolution ill-posedness and also to address noise issues in the image. The program’s iteration exit point is determined using the L ² error norm

$$ \frac{\|u^{(n)}-u^{(n-1)}\|_{2}^{2}}{\|u^{(n)}\|_{2}^{2}}<\varepsilon, $$

((13))

Fig. 3

Flow diagram for the proposed framework

where 0<ε<1 is a tuning constant that determines the final error of the results.

2.2.2 2.2.2 Physical significance and roles of α(x)

The variable, α∈[1,2], sweeps values between one and two according to the local features of an image (Fig. 4). From (11), we observe the following cases:

• 1. In flat regions (|∇(G _σ ∗f)|→0), α=1. Substituting this value of α into (12) yields

  $$ \begin{aligned} \frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})\\ &+\text{div}\left(\frac{2+\frac{|\nabla u|}{\beta}}{1+\frac{|\nabla u|}{\beta}}\nabla u\right)\\&-\lambda(u-f). \end{aligned} $$

  Now, expanding the divergence part of this equation produces

  $$ \begin{aligned} \frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})\\ &+\text{div}\left(\frac{1}{1+\frac{|\nabla u|}{\beta}}\nabla u\right)+\triangle u, \end{aligned} $$

  ((14))

  Fig. 4

  

  Variation of the proposed diffusivity with respect to different values of α(x)

  which combines two regularizing models, namely ROF and isotropic diffusion. Therefore, the formulation can isotropically remove noise in these (flat) regions. Additionally, if β is carefully tuned, we may preserve weak edges due to the presence of a regularizing component—middle term of (14)—that is similar to that of TV.

• 2. Near edges (|∇(G _σ ∗f)|→∞), α=2. Thus, Eq. 12 becomes

  $$\begin{aligned} \frac{\partial u}{\partial t}=&\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})\\&+\text{div}\left(\frac{2+\frac{|\nabla u|}{\beta}}{\left(1+\frac{|\nabla u|}{\beta}\right)^{2}}\nabla u\right)\\ &-\lambda(u-f), \end{aligned} $$

  which contains a regularizing part proposed by Ogada et al. [43]. This formulation helps to suppress noise, avoid diffusion of edges, and enhance the spatial resolution of an image.

• 3. From the above two cases, I and II, we see that α(x) plays another role of segmenting an image into two subregions, namely Ω ₁ (flat regions; α=1) and Ω ₂ (edges and contours; α=2). Hence, α is an edge-defining variable. An important aspect of α is that it contains a convolution operator between the Gaussian kernel and the image, G _σ ∗f, which helps to suppress noise and other unwanted artifacts—thus detecting useful image features robustly, even under harsh imaging conditions (Figs. 5 and 6).

  Fig. 5

  

  Edge maps of a synthetic image: a original and noisy (white Gaussian noise) image, b image generated from the Ogada et al. model, and c image generated using our method

  Fig. 6

  

  Edge maps of an elephant image: a original and noisy (white Gaussian noise) image, b image generated from the Ogada et al. model, and c image generated using our method

2.2.3 2.2.3 Properties of the model

To comprehensively understand the (mechanical) properties of our model, consider two orthogonal vectors, T(x)=(u _x ,u _y )/|∇u| and N(x)=(−u _y ,u _x )/|∇u|, in an image, u, where u _x and u _y are the first-order partial derivatives of u in the x- and y-direction, respectively, and |∇u|≠0. Additionally, let $u_{_{\text {TT}}}$ (tangential) and $u_{_{\text {NN}}}$ (normal) be the second-order partial derivatives of u, representing diffusions, in the directions of T and N, respectively. Defining $u_{_{{\text {TT}}}}$ and $u_{_{\text {NN}}}$ as

$$\begin{aligned} {}\!\!u_{_{\text{TT}}}&=T^{\prime}\nabla^{2}uT=\!\frac{1}{|\nabla u|^{2}}\left({u_{x}^{2}}u_{yy}+{u_{y}^{2}}u_{xx}-2u_{x}u_{y}u_{xy}\right)~\text{\!and} \\ {} u_{_{\text{NN}}}&=N^{\prime}\nabla^{2}uN\!=\frac{1}{|\nabla u|^{2}}\left({u_{x}^{2}}u_{xx}+{u_{y}^{2}}u_{yy}+2u_{x}u_{y}u_{xy}\right), \end{aligned} $$

where $T^{\prime }=(1/|\nabla u|) \left (\begin {array}{c}u_{y}\\ u_{x}\end {array}\right), N^{\prime }=(1/|\nabla u|) \left (\begin {array}{c}u_{x}\\ u_{y}\end {array} \right),$and $\nabla ^{2}u=\left (\begin {array}{cc}u_{\textit {xx}} & u_{\textit {xy}}\\ u_{\textit {xy}} & u_{\textit {yy}} \end {array}\right)$ is the Hessian matrix; Eq. 12 can compactly be re-written in terms of $u_{_{\text {TT}}}$ and $u_{_{\text {NN}}}$ as

$$ {\fontsize{7.0pt}{9.6pt}\selectfont{\begin{aligned} {}\frac{\partial u}{\partial t}&=\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}(W_{k}B_{k}D_{k}u-y_{k})+\left(\frac{2+\frac{|\nabla u|}{\beta}}{\left(1+\frac{|\nabla u|}{\beta}\right)^{\alpha\text{(x)}}}\right)u_{_{\text{TT}}} \\ &\quad+\left(\frac{\left(1+\frac{|\nabla u|}{\beta}\right)\left(\left(2+\frac{|\nabla u|}{\beta}\right)+1\right)-\left(2+\frac{|\nabla u|}{\beta}\right)\alpha(\mathrm{x})}{\left(1+\frac{|\nabla u|}{\beta}\right)^{\alpha(\mathrm{x})+1}}\right)u_{_{\text{NN}}}-\lambda(u-f). \end{aligned}}} $$

((15))

Equation 15 possesses some interesting properties worth noting: in flat regions (|∇u|→0 and α→1), the equation reduces to

$$ {\fontsize{8.8pt}{9.6pt}\selectfont{\begin{aligned} {}\frac{\partial u}{\partial t}\,=\,\frac{1}{M}\!\sum_{k=1}^{M}\!W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}\!(\!W_{k}B_{k}D_{k}u-y_{k}\!)+(\!{Cu}_{_{\text{TT}}}+u_{_{\text{NN}}})-\lambda(\!u-f), \end{aligned}}} $$

((16))

(C is a constant value) which contains an isotropic diffusion component ($\triangle u=u_{_{\text {TT}}}+u_{_{\text {NN}}}$ is the pure heat equation) that removes noise uniformly over the regions. Also, near edges (|∇u|→∞ and α→2), the coefficient of $u_{_{\text {NN}}}$—which contains the denominator larger than that of $u_{_{\text {TT}}}$—vanishes faster. Consequently, the $u_{_{\text {TT}}}$ component that is responsible to preserve edges dominates.

2.3 Numerical implementation

The desire to implement our method using an explicit scheme is attributed to the following reasons: computational efficiency, ability to produce more accurate and appealing results, intuitiveness to be understood and analyzed mathematically, and stability over the time interval defined by the Courant-Friedrichs-Lewy criterion (0<τ≤0.25) [48]. A significant drawback, however, of explicit schemes is that they are susceptible to instabilities for larger iteration steps.

Now, consider a five-point discretized space with four directions: north, south, east, and west (Fig. 7). The gradients of u _i,j along these directions are, respectively, $\triangle _{i,j}^{N}=u_{i,j+1}-u_{i,j}, \triangle _{i,j}^{S}=u_{i,j-1}-u_{i,j}, \triangle _{i,j}^{E}=u_{i+1,j}-u_{i,j}, \text {~and~} \triangle _{i,j}^{W}=u_{i-1,j}-u_{i,j}$, and the corresponding discrete conduction coefficients from the divergence part of (12) are

$${}{cN}_{i,j}=\left(\frac{2+\frac{|\triangle_{i,j}^{N}|}{\beta}}{\left(1+\frac{|\triangle_{i,j}^{N}|}{\beta}\right)^{\alpha_{i,j}}}\right), {cS}_{i,j}=\left(\frac{2+\frac{|\triangle_{i,j}^{S}|}{\beta}}{\left(1+\frac{|\triangle_{i,j}^{S}|}{\beta}\right)^{\alpha_{i,j}}}\right), $$

$${}{cE}_{i,j}=\left(\frac{2+\frac{|\triangle_{i,j}^{E}|}{\beta}}{\!\left(1+\frac{|\triangle_{i,j}^{E}|}{\beta}\right)^{\alpha_{i,j}}}\!\right)\!,\!\! \text{~~and~~}{cW}_{i,j}=\left(\frac{2+\frac{|\triangle_{i,j}^{W}|}{\beta}}{\!\left(1+\frac{|\triangle_{i,j}^{W}|}{\beta}\right)^{\alpha_{i,j}}}\!\right)\!, $$

where 0≤i≤P and 0≤j≤Q; P and Q are, respectively, the horizontal and vertical dimensions of u _i,j . Following the explicit scheme, we discretize the divergence component of our formulation as

$$\text{div}_{i,j}={cN}_{i,j}\triangle_{i,j}^{N} + {cS}_{i,j}\triangle_{i,j}^{S} + {cE}_{i,j}\triangle_{i,j}^{E} + {cW}_{i,j}\triangle_{i,j}^{W}. $$

Fig. 7

Explicit numerical scheme

From the given discretizations, therefore, the steepest descent implementation of the regularized super-resolution method in (12) becomes

$$ {\fontsize{9.1pt}{9.6pt}\selectfont{\begin{aligned} {}u^{(n+1)}_{i,j}=u^{(n)}_{i,j}-\tau\left(R\left(u_{i,j}^{(n)}\right)-M\mu\text{div}^{(n)}_{i,j}+M\mu\lambda\left(u_{i,j}^{(n)}-f_{i,j}\right)\right)\!, \end{aligned}}} $$

((17))

where μ is a tuning constant and

$$ R\left(u_{i,j}^{(n)}\right)=\frac{1}{M}\sum_{k=1}^{M}W_{k}^{\prime}B_{k}^{\prime}D_{k}^{\prime}\left(W_{k}B_{k}D_{k}u_{i,j}^{(n)}-(y_{k})_{i,j}\right) $$

((18))

is a super-resolution result at the nth iteration. The boundary conditions of (17) are as follows: $u_{i,j}^{(0)}=(y_{i,j})_{1}=y_{1}(ih,jh), u_{i,0}^{(n)}=u_{i,1}^{(n)},~u_{0,j}^{(n)}=u_{1,j}^{(n)},~u_{P,j}^{(n)}=u_{P-1,j}^{(n)},~\text {and} ~u_{i,Q}^{(n)}=u_{i,Q-1}^{(n)}$

2.4 Performance evaluation of the super-resolution models

To quantitatively evaluate the performance of different super-resolution models, we have used three metrics: peak-signal-to-noise ratio (PSNR) [49, 50], edge similarity (ESIM) [45], and mean structure similarity (MSSIM) [51]. The PSNR measures signal strength relative to noise in the image and is defined by the equation

$$ \text{PSNR}=10\log_{10}\left(\frac{255^{2}PQ}{\|u-f\|_{2}^{2}}\right). $$

((19))

Though widely applied in many image-processing disciplines, PSNR fails to explain the quality of edges in the restored scenes. Guo et al. addressed this limitation by proposing the metric

$$ \text{ESIM} = \widetilde{\text{PSNR}}(\text{EM}(u), \text{EM}(f))~~\text{[45]}, $$

((20))

where EM(u) and EM(f) are, respectively, the edge maps of u (restored image) and f (initial ideal image), and

$$\widetilde{\text{PSNR}}=10\log_{10}\frac{PQ|\max(f)-\min(f)|^{2}}{\|u - f\|_{2}^{2}} $$

is as proposed in [52]. Moreover, both PSNR and ESIM inadequately address the perceptual qualities of scenes. Wang and colleagues proposed a metric (MSSIM) that explains the statistical inter-dependency of pixels in scenes, thus quantifying their visual appeals [51]. The MSSIM is governed by the equation

$$ \text{MSSIM}=\frac{(2\mu_{u}\mu_{f}+c_{1})(2\sigma_{uf}+c_{2})}{\left({\mu_{u}^{2}}+{\mu_{f}^{2}}+c_{1})({\sigma_{u}^{2}}+{\sigma_{f}^{2}}+c_{2}\right)}, $$

((21))

where the variables, respectively defined for u and f are as follows: μ _u and μ _f , mean; ${\sigma _{u}^{2}}$ and ${\sigma _{f}^{2}}$, variance; and σ _uf , covariance. And c ₁ and c ₂ are stabilizing constants.

3 Experiments

We conducted experiments from both simulated and real environments to test and compare the performance of the proposed method and some classical super-resolution methods, namely (Total variation) [33], (Spatially weighted total variation) [42], (Fractional order total variation) [41], and (Adaptive total variation) [40]. In the first experiment, we degraded each of the original high-resolution images of Boat, Bridge, Building, Fish, Goldhill, Lena, Mandrill, and Wheel (http://decsai.ugr.es/cvg/CG/base.htm) (Fig. 8) to generate the corresponding sequence of ten low-resolution images. Next, we applied TV-SR, SWTV-SR, FTV-SR, ATV-SR, and the new method to restore the respective original versions of the degraded image sequences (Fig. 9). These procedures were repeated in the second experiment but with the input images being the degraded low-resolution video frames of EIA (https://users.soe.ucsc.edu/~milanfar/software/sr-images.html) (Fig. 10). The last experiment tested the efficacy of our method for two conditions of the variable exponent, α(x): fixed (α=1 and 2) and adaptive (Fig. 11).

Fig. 8

Original high-resolution images: a Boat, b Bridge, c Building, d Fish, e Goldhill, f Lena, g Mandrill, and h Wheel

Fig. 9

Simulated super-resolution results of different methods

Fig. 10

Super-resolution results of different methods for real video sequences: a low-resolution, b TV-SR, c SWTV-SR, d FTV-SR, e ATV-SR, and f our method

Fig. 11

Super-resolution results of our method for various conditions of the variable exponent, α(x): a original, b low-resolution, c α=1, d α=2, and e α(x)

4 Results and discussions

The visual results demonstrate that the proposed method outperforms in several cases compared with some state-of-the-art classical methods (Figs. 9 and 10). Both simulated and real experiments show that the new approach generates appealing images that are sharper and detailed. Other methods, such as TV-SR, SWTV-SR, and FTV-SR, reveal obvious artifacts—ringing, blocking, and staircasing (Fig. 9). The last experiment proves that the adaptive nature of our formulation helps to preserve useful image features and suppress noise; setting constant the edge-locating variable exponent, α(x), in the proposed model lowers its performance (Fig. 11). We observed earlier in Section 2.2.2—“Physical significance and roles of α(x)”—that fixing α to 1 (α=1), for example, makes the super-resolution problem regularized by both TV and isotropic diffusion, and this promotes edge recovery and noise removal, respectively. The results in the third experiment prove this mathematical intuition but also reveal some artifacts that are probably due to fixing α(x), as depicted by Fig. 11 c. In Fig. 11 e, we observe that adaptively updating α(x) makes the results appear attractive.

Quantitative results further confirm that the proposed model is superior as it achieves higher quality values. Table 1 shows that the new method achieves promising PSNR values in most cases, which implies that the method retains higher signal contents in the restored results. In terms of the edge recovery capabilities, our method performs better as it shows larger values of ESIM compared with those from the other methods (Table 2). This finding concurs with the subjective assessments that the proposed method generates images with sharper and stronger edges, which raises the value of the ESIM. In Table 3, the results demonstrate that the new method produces MSSIM values that are larger than those produced by TV-SR, SWTV-SR, FTV-SR, and ATV-SR. This quantitative quality assessment justifies the observation that our results are visually appealing (Figs. 9 and 10). Recall that MSSIM favors the human visual system. Furthermore, Table 4 demonstrates that our method offers lower computational times—thus making it a suitable choice in tasks that demands parallel computing.

Table 1 Peak signal-to-noise ratio (PSNR) measurements of different super-resolution methods

Table 2 Edge similarity (ESIM) measurements of different super-resolution methods

Table 3 Mean structural similarity (MSSIM) measurements of different super-resolution methods

Table 4 Algorithmic CPU times (in seconds) of different super-resolution methods

The performance of ATV-SR closely follows that of our method. Subjectively, the results of these two methods are hard to distinguish (Fig. 9; images of Building and Lena; and Fig. 10). However, numerical results indicate that the proposed approach outperforms in several cases (Tables 1, 2, and 3). For the “Wheel” image, however, Table 1 shows that the ATV-SR outperforms our method by a small amount. Also, the ATV-SR is slightly faster than the proposed method (Table 4), but the deviation is too small that we may assume the two methods perform equally.

Despite the promising performance, the proposed method suffers from one weakness—it tends to slightly blur the output images. This is probably caused by the low-pass filtering operation of the regularizing functional or inappropriate estimation of the blur function. More research is thus needed to address the limitation. It is worth noting that the super-resolution results are, however, not only limited to human consumption but also to industrial applications, such as control and automation [53, 54], object detection, and feature extraction. With the proposed method generating promising objective results, we hope that it may as well suit these other disciplines.

5 Conclusions

In this paper, we have presented an adaptive multi-frame super-resolution model that sufficiently restores fine image details. The new method incorporates a spatially varying regularizing term that updates its value according to the local image features—linear isotropic in flat regions and nonlinear anisotropic near edges. This flexibility and adaptability makes the model generate promising results, objectively and quantitatively. Also, the proposed adaptive term includes a convolution operation with the Gaussian filter, and this allows the model to robustly emphasize critical and meaningful features. Experimental results visually demonstrate the strength of the new method that it reveals more information in the reconstructed images compared with other methods. Objectively, we have shown that the method generates promising values of the quality metrics (PSNR, ESIM, and MSSIM).

In the future, we are contemplating the possibilities of extending our method to other fields, such as in medical imaging. For example, doctors in sonography require high-quality ultrasound images to provide accurate treatments to patients. The current instruments produce low-quality images that are heavily degraded by multiplicative noise. The problem can be approached in a variety of ways. In the context of the new method, three important processes that may help to address the problem are (1) modifying the prior to cover multiplicative noise, (2) transforming the model into the three-dimensional space to comprehensively treat the ultrasound images, and (3) implementing the model using more accurate and fast numerical schemes that support real-time parallel computing.