# Weighted gradient domain image processing problems and their iterative solutions

- Jung Gap Kuk
^{1}and - Nam Ik Cho
^{2}Email author

**2013**:7

https://doi.org/10.1186/1687-5281-2013-7

© Kuk and Cho; licensee Springer. 2013

**Received: **7 June 2012

**Accepted: **22 November 2012

**Published: **23 January 2013

## Abstract

This article explores an energy function and its minimization for the weighted gradient domain image processing, where variable weights are applied to the data term of conventional function for attaining better results in some applications. To be specific, larger weights are given to the regions where original pixel values need to be kept unchanged, like strong edge regions in the case of image sharpening application or high contrast regions when fusing multi-exposure images. In the literatures, it is shown that the solution to a constant weight problem can be efficiently obtained in the frequency domain without iterations, whereas the function with the varying weights can be minimized by solving a large sparse linear equation or by iterative methods such as conjugate gradient or preconditioned conjugate gradient (PCG) methods. In addition to introducing weighted gradient domain image processing problems, we also proposed a new approach to finding an efficient preconditioning matrix for this problem, which greatly reduces the condition number of the system matrix and thus reduces the number of iterations for the PCG process to reach the solution. We show that the system matrix for the constant weight problem is an appropriate preconditioner, in the sense that a sub-problem in the PCG is efficiently solved by the FFT and also it ensures the convergent splitting of the system matrix. For the simulation and experiments on some applications, it is shown that the proposed method requires less iteration, memory, and CPU time.

## Keywords

## 1 Introduction

Since human visual system (HVS) is sensitive to the intensity changes, processing an image in the gradient domain often produces subjectively better results than the conventional intensity domain processing. Specifically, the gradient domain approach has been successfully applied to high dynamic range (HDR) imaging [1], image stitching [2–5], filtering [6], alignment [7], matching [8], etc. In the case of image stitching problems which include seamless cloning [2, 5] and image composite for panoramic view [3, 4], the gradient domain processing is considered the state-of-the-art method.

**g**(

*x*,

*y*), the image that best matches

**g**(

*x*,

*y*) is found by minimizing the energy function

*u*(

*x*,

*y*) is the gradient of the output image

*u*(

*x*,

*y*). Recently in [10], an energy function in the form of data term plus gradient term is also considered, i.e.,

where *d*(*x*,*y*) is a data function which is usually the input image, and *λ* controls the balance between the terms. The data term has the role of keeping an output close to the input (in order not to be deviated from the input too much), while the gradient term matches image gradients to be close to the desired one. The analysis in
[10] shows that the solution should satisfy screened Poisson equation which is well known in physics, and the equation can be solved directly (not iteratively) in the frequency domain. By using this formulation, a number of image processing applications can be handled such as sharpening, image stitching, and deblocking.

where the data weight *w*(*x*,*y*) ranges from 0 to 1. By controlling the weights depending on regional properties, we later show that a visually better result is obtained in some applications. The basic idea is to give larger weights to the pixels in noticeable regions where the edge is strong or contrast is high, so that well captured regions are kept intact and others are changed according to the manipulated gradients. Although the conventional iterative methods such as conjugate gradient (CG) or preconditioned conjugate gradient (PCG) methods can be used to obtain the solution of this general problem (varying weights)
[11], it is noted that they require a large amount of memory and long computation time. Hence, we analyze the problem and derive a more efficient solver to this problem. In the analysis, we first show that the solution minimizing (3) should satisfy a variant of screened Poisson equation which we call *inhomogeneously* screened Poisson equation. Then, we solve the equation based on PCG where the system matrix of the screened Poisson equation is selected as a preconditioner. Since this solver obviates the need of explicit form of large linear system, it requires less memory space. We also show that this preconditioner implies the convergent splitting of system matrix. With the simulated and real data, we demonstrate that the proposed solver is faster than the PCG. Finally we show some examples of weighted gradient domain image processing, which provides better subjective/objective quality than the intensity domain solution and also the gradient domain solution without weights.

This article is organized as follows. The inhomogeneously screened Poisson equation is derived in Section 2.1 and a solution based on the PCG is presented in Section 2.2. The convergence rate is also analyzed in Section 2.3. In Section 3, the proposed formulation is applied to the image sharpening problem (Section 3.1) and exposure fusion problem (Section 3.2). Finally in Section 4, conclusions are given.

## 2 Variational formulation and its solution

In this section, we derive the inhomogeneously screened Poisson equation from the energy function in (3), which includes spatially varying data weight. Then we present an iterative solver for the equation and analyze its convergence rate.

### 2.1 Inhomogeneously screened Poisson equation

where *i* and *j* denote pixel indices, *V* is a set of all the pixels, *N*
_{
i
} is a set of 4 neighboring pixels of the *i* th pixel, and *u*
_{
i
}, *d*
_{
i
}, and *g*
_{
i
j
} are discrete counterparts of *u*(*x*,*y*), *d*(*x*,*y*), and **g**(*x*,*y*), respectively.

### 2.2 Iterative solution

where *W* is a diagonal matrix with *w*
_{
i
} on the diagonal, **b** is a vector whose *i* th element is
${w}_{i}{d}_{i}-\sum _{j\in {N}_{i}}{g}_{\mathit{\text{ji}}}$ and *L* is a 5-point Laplacian matrix. It is noted that *W*−*λ* *L* in (8) is a large, sparse, and symmetric matrix, and it can be directly solved by triangular factorization. However, direct solver needs very large memory, even though the sparsity of the matrix is fully exploited
[12]. Hence, an iterative method is usually preferred to the direct method when solving this kind of problem, such as the preconditioned conjugate gradient (PCG). Preconditioning is to transform the original problem *A* **u**=**b** to the one that can converge faster, by pre-multiplying an inverse of a certain matrix that reduces the condition number (ratio of maximum to minimum eigenvalue in the case of symmetric matrix) of the resulting system. To be specific, the inverse of a preconditioning matrix *M* is multiplied to (8), i.e., *M*^{−1}
*A* **u**=*M*^{−1}
**b**, where *A*=*W*−*λ* *L*, so that the eigenvalue spread of *M*^{−1}
*A* is smaller than that of original system matrix *A*. In summary, the PCG is to find a matrix *M* that reduces the condition number of the linear system, and then iterate the conjugate gradient steps as in the Algorithm 1.

### Algorithm 1 PCG steps

*Initialization*:

**u**
_{0}=**0**,**r**
_{0}=**b**−*A* **u**
_{0}, **t**
_{0}=*M*^{−1}
**r**
_{0}, **p**
_{0}=**t**
_{0}

*Loop*:

- 1.
$a=\frac{{\mathbf{r}}_{i}^{T}{\mathbf{t}}_{i}}{{\mathbf{p}}_{i}^{T}A{\mathbf{p}}_{i}}$

- 2.
**u**_{ i+1}=**u**_{ i }+*a***p**_{ i } - 3.
**r**_{ i+1}=**r**_{ i }-*a**A***p**_{ i } - 4.
**t**_{ i+1}=*M*^{-1}**r**_{ i+1} - 5.
$\xdf=\frac{{\mathbf{r}}_{i+1}^{T}{\mathbf{t}}_{i+1}}{{\mathbf{r}}_{i}^{T}{\mathbf{t}}_{i}}$

- 6.
**p**_{ i+1}=**t**_{ i }+*ß***p**_{ i }

*M*. But there is no general way to design the optimal

*M*, i.e., the design of the preconditioner is problem-dependent [11]. Though, there are two well known requirements to be a good preconditioner. First, the preconditioning matrix should easily be inverted. In other words, a linear system involving

*M*(step 4 of Algorithm 1) should be easily solved, because it is the main problem in the loop. Second, when the system matrix

*A*is written as

*M*−

*N*where

*M*is a non-singular matrix, the splitting should be a convergent splitting, which makes the solver stable [13]. Note that the representation

*M*−

*N*of

*A*is called a convergent splitting when

*M*is non-singular and

*ρ*(

*M*

^{−1}

*N*)<1, where

*ρ*(

*Θ*) measures the spectral radius of the matrix

*λ*, defined as

where *σ*(*Θ*) denotes the spectrum of *λ*, that is, the set of eigenvalues of *λ*.

*W*−

*λ*

*L*into the matrices which represent the screened Poisson equation and the residual. More formally,

*W*−

*λ*

*L*is written as

*I*is the identity matrix and (

*I*−

*λ*

*L*) is a system matrix that represents screened Poisson equation for the minimization of (2) without the weights. From the splitting in (10), we choose to use (

*I*−

*λ*

*L*) as a preconditioner, which will be denoted as

*A*

_{ s }in the rest of this article. This preconditioner satisfies the required constraints stated above: first,

*A*

_{ s }is easily inverted, or the linear system

*A*

_{ s }

**u**=

**b**is efficiently solved because the optimal solution of the linear system

*A*

_{ s }

**u**=

**b**is the same as that of screened Poisson equation [10] which can be directly solved in the frequency domain. Also, the proposed PCG is memory-efficient because we do not need the explicit form of the system matrix

*A*

_{ s }when solving the linear system

*A*

_{ s }

**u**=

**b**. Furthermore, we have the advantage in the implementation issue because there are many efficient libraries for fast real transform such as FFTW [14] and Intel Integrated Performance Premitive [15] for solving

*A*

_{ s }

**u**=

**b**. Second, the splitting in (10) is a convergent splitting, that is,

*A*

_{ s }is non-singular and $\rho \left({A}_{s}^{-1}\right(I-W\left)\right)<1$. It is easy to show that

*A*

_{ s }is non-singular because all the eigenvalues of

*A*

_{ s }is larger than 1. More precisely,

*A*

_{ s }can be written as

where *U*^{−1}
*Γ* *U* is a decomposition of the Laplacian matrix *L* and *Γ* is a diagonal matrix whose *i* th diagonal element *γ*
_{
i
} denotes the eigenvalue of Laplacian matrix. Since the Laplacian matrix is a negative semi-definite matrix (*γ*
_{
i
}≤0 for all *i*), it follows that 1−*λ* *γ*
_{
i
}≥1, where 1−*λ* *γ*
_{
i
} is the *i* th eigenvalue of *A*
_{
s
}. Hence, all the eigenvalues of *A*
_{
s
} is larger than 1. The other requirement for the convergence splitting, i.e.,
$\rho \left({A}_{s}^{-1}\right(I-W\left)\right)<1$, can also be proved as follows.

#### Proof

*W*−

*λ*

*L*and

*I*−

*W*as

*A*and

*N*, respectively. Supposing that

*α*is an eigenvalue of ${A}_{s}^{-1}N$ and

**x**is a corresponding eigenvector to

*α*, we can write ${A}_{s}^{-1}N\mathbf{x}=\alpha \mathbf{x}$. Pre-multiplying

**x**

^{ T }to both sides of the equation, we have

*A*

_{ s }with

*A*+

*N*and rearranging give

Since both *A* and *N* are positive semi-definite and *α*≠1,
$\frac{\alpha}{1-\alpha}\ge 0$, i.e., 0≤*α*<1. Hence
$\rho \left({A}_{s}^{-1}N\right)<1$. □

### 2.3 Convergence analysis

In this section, we analyze the convergence rate of the proposed preconditioner with the simulated data and real problems. We consider three measures for the evaluation of convergence rate, namely condition number, elapsed time and the amount of required memory. Specifically, the proposed method is compared with standard conjugate gradient without preconditioning, and several incomplete LU (ILU) factorization methods for the sparse symmetric system such as ILU0 [11], modified ILU (MILU) [11] and modified Crout variant of ILU (MILUC) [16].

**u**

_{ o }, and measure how fast the

**u**

_{ i }in Algorithm 1 converges to

**u**

_{ o }. Specifically, a linear system (8) is built where the diagonal element

*w*

_{ i }of the matrix

*W*is randomly generated within the range of [0 1], and

*λ*is set to be 30. We need not specifically define

*d*(

*x*,

*y*) and

**g**(

*x*,

*y*), because we need just

**b**that corresponds to

**u**

_{ o }, which is computed as

**b**=(

*W*−

*λ*

*L*)

**u**

_{ o }. Given this linear system, we compare the condition numbers in Table 1 for the matrices of moderate sizes ranging from 100×100 to 2500×2500. In this table, the density of nonzero elements of MILUC is given in the parenthesis, which is the ratio of the number of nonzero elements of MILUC to that of the original matrix. It can be seen that the proposed preconditioning method has the smallest condition number. The linear system with the lower condition number converges faster, which is also verified by plotting the log RMS error curve for the 2500×2500 case in Figure 1. The log RMS error is defined as $\underset{10}{log}\left(\sqrt{|{\mathbf{u}}_{i}-{\mathbf{u}}^{o}|/N}\right)$, where

**u**

_{ i }denotes the solution after the

*i*th iteration, and

*N*is the number of whole pixels.

**Condition numbers for a**
p
**×**
p
**matrix**

p | CG | ILU0 | MILU | MILUC | Proposed |
---|---|---|---|---|---|

100 | 602.9 | 53.5 | 7.4 | 2.6(2.14) | 1.8 |

400 | 597.1 | 54.7 | 8.4 | 2.7(2.24) | 1.7 |

900 | 639.0 | 59.2 | 9.0 | 2.7(2.27) | 1.8 |

1600 | 642.5 | 59.8 | 8.9 | 2.8(2.29) | 1.8 |

2500 | 635.9 | 59.5 | 8.8 | 2.7(2.30) | 1.8 |

^{−3}, and also the amount of required memory. In this implementation, the size of the linear system is set to be 250000×250000. Note that all the steps in Algorithm 1, except for the step 4, are equally implemented for all the methods. The step 4 in our problem is solved by using FFTW as stated previously. Table 2 shows the result that the amount of required memory space and CPU time for the proposed solver are less than those of others. It is found that the elapsed time of MILUC is much larger than others because it spends much time on the incomplete factorization of denser nonzero pattern. The ratio of the number of nonzero elements of MILUC to that of original matrix is also given in the parenthesis.

**Required memory space (KB), elapsed time (**
ms
**) and required iterations to reach the error of 10**
^{
−3
}
**for a 250000×250000 matrix**

CG | ILU0 | MILU | MILUC | Proposed | |
---|---|---|---|---|---|

mem | 32,6842 | 62,624 | 46,984 | 81,228 | 25,444 |

time | 2.9661 | 0.8661 | 0.3201 | 872.56 | 0.2110 |

iters | 62 | 20 | 7 | 4(2.33) | 3 |

^{−3}) as observed in Figure 2. The comparison for the exposure fusion example is given in Table 4 where an additional experiment is conducted for the reduced image size (250×150), because MILUC causes an out-of-memory problem for the actual image size (1025×769). As shown in Table 4, the proposed method also shows better performance than the others. The convergence error curve for this problem is plotted in Figure 3.

**Required memory space (KB), elapsed tie (**
ms
**) and number of required iterations to reach the error of 10**
^{
−3
}
**for the image sharpening example**

CG | ILU0 | MILU | MILUC | Proposed | |
---|---|---|---|---|---|

mem | 16,396 | 24,770 | 36,652 | 43,128 | 13,272 |

time | N/M | 0.6596 | 0.1328 | 20.4474 | 0.1035 |

iters | N/M | 135 | 28 | 17(2.32) | 10 |

**Required memory space (KB), elapsed time**
(ms
**) and number of required iterations to reach the error of 10**
^{
−3
}
**for the HDR imaging problem**

Image size | CG | ILU0 | MILU | MILUC | Proposed | |
---|---|---|---|---|---|---|

1025×769 | mem | 163,688 | 255,572 | 241,785 | N/A | 110,488 |

time | N/M | 53.9044 | 6.2435 | N/A | 5.5096 | |

iters | N/M | 384 | 43 | N/A | 17 | |

250×150 | mem | 4,012 | 5,428 | 5,768 | 13,232 | 3,528 |

time | N/M | 0.4574 | 0.0936 | 11.3748 | 0.0516 | |

iters | N/M | 148 | 26 | 15(2.32) | 8 |

## 3 Examples of weighted gradient domain image processing

In this section, we present two examples, image sharpening and gradient domain exposure fusion, where the gradient domain processing with the variable weights can be more effective than the existing approaches.

### 3.1 Image sharpening

Image sharpening is one of the most commonly used techniques for enhancing the contrast. The conventional method for image sharpening is the Laplacian subtraction approach, i.e., a Gaussian blur is applied to a given image, the blurred image is subtracted from the original to form a contrast image, and then the contrast image is weighted and added to the original. In this procedure, since the subtraction of blurred image from the original can be interpreted as an approximation of Laplacian filtering, this technique is called Laplacian substraction.

*d*(

*x*,

*y*) is the input image and

**g**(

*x*,

*y*) is designed such that the gradients are boosted as

where *c*
_{1}(>1) and *c*
_{2} are constants, and **s**(*x*,*y*) is a vector image containing the saliency in both *x* and *y* directions around the pixel coordinate (*x*,*y*). According to the gradient manipulation strategy in (12), all the gradients are basically boosted by constant value *c*
_{1} and they are selectively magnified again by the edge saliency **s**(*x*,*y*). The parameter *c*
_{2} controls the amount of sharpening boosted by **s**(*x*,*y*). Although the gradient domain approach seems to be different from the Laplacian subtraction method, it has been shown that this approach can be interpreted as a generalized version of Laplacian subtraction method
[10].

where *g*
_{
i
} is the gradient magnitude given by
$\sqrt{{d}_{\mathit{\text{iv}}}^{2}+{d}_{\mathit{\text{ih}}}^{2}}$ with *d*
_{
i
v
} and *d*
_{
i
h
} being the vertical and horizontal gradient value, respectively, and *σ*
_{
d
} controls attenuation. Note that the gradient manipulation term in our solution is designed in the same manner as in (12).

### 3.2 Weighted gradient domain exposure fusion

*d*(

*x*,

*y*),

*w*(

*x*,

*y*), and

*g*(

*x*,

*y*). First, the data function

*d*(

*x*,

*y*) is devised to keep the regions with better contrast in either of under or over exposed images, i.e.,

*I*

_{ u }(

*x*,

*y*) and

*I*

_{ o }(

*x*,

*y*) denote under- and over-exposed image, respectively, and

*C*(

*I*(

*x*,

*y*)) measures the contrast of the image

*I*around pixel coordinate (

*x*,

*y*). However, such hard decision scheme in (14) would fail when the measured contrast is low. In other words, the reliability of the data function becomes low when the measured contrast is low. We handle this problem by designing the data weight function

*w*(

*x*,

*y*) in a spatially varying manner. To be specific,

*w*(

*x*,

*y*) is designed such that the weight is close to one when the contrast is strong and to zero when the contrast is weak due to saturation, in the form of

*C*(

*x*,

*y*)= max{

*C*(

*I*

_{ u }(

*x*,

*y*),

*C*(

*I*

_{ o }(

*x*,

*y*)} and

*σ*controls the attenuation. The target gradient function

*g*(

*x*,

*y*) is designed such that the larger gradient is preferred as

where *ϕ*(·) is a weight function which attenuates large gradients and magnifies small gradients, as defined in
[1].

## 4 Conclusions

We have presented weighted gradient domain image processing problems where spatially varying weights are applied to the data term of conventional energy function. The problem is formulated as a system of linear equation, which is more efficiently solved by iterative methods such as PCG than by direct inverse or factorization methods. For solving the problem with the PCG, the most important thing is to find a preconditioning matrix that makes the system matrix have low condition number. We have shown that the system matrix for the constant weight problem is an appropriate choice for the preconditioner. Specifically, a subproblem in the PCG steps is efficiently solved by the proposed preconditioning matrix and it is also shown that this matrix induces the convergent splitting of the system matrix. Simulation and experiments also show that the proposed method converges faster than the existing methods, and also requires less memory space and CPU time. Applications of the weighted gradient domain image processing problems have also been presented, with the examples of image sharpening and the exposure fusion problems. In the comparison results, it is shown that better outputs are produced owing to the spatially varying design of data weight.

## Declarations

### Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2012-0000913).

## Authors’ Affiliations

## References

- Fattal R, Lischinski D, Michael W: Gradient domain high dynamic range compression.
*ACM Trans. Graph*2002, 21(3):249-256.View ArticleGoogle Scholar - Pérez P, Gangnet M, Blake A: Poisson image editing.
*ACM Trans. Graph*2003, 22(3):313-318. 10.1145/882262.882269View ArticleGoogle Scholar - Levin A, Zomet A, Peleg S, Weiss Y: Seamless image stitching in the gradient domain.
*Lecture Notes Comput. Sci*2004, 3024: 377-389. 10.1007/978-3-540-24673-2_31View ArticleGoogle Scholar - Agarwala A: Efficient gradient-domain compositing using quadtrees.
*ACM Trans. Graph*2007, 26(3):94-98. 10.1145/1276377.1276495View ArticleGoogle Scholar - Jia J, Sun J, Tang CK, Shum HY: Drag-and-drop Pasting.
*ACM Trans. Graph*2006, 25(3):631-637. 10.1145/1141911.1141934View ArticleGoogle Scholar - Bhat P, Zitnick CL, Cohen M, Curless B: GradientShop: a gradient-domain optimization framework for image and video filtering.
*ACM Trans. Graph*2010, 29(2):1-14.View ArticleGoogle Scholar - Baker S, Matthews I: Lucas-Kanade 20 years on: a unifying framework.
*Int. J. Comput. Vis*2004, 56(3):221-255.View ArticleGoogle Scholar - Wu Y, Fan J: Contextual flow. In
*CVPR’09*. Miami, FL; 2009:33-40.Google Scholar - Sun J, Xu Z, Shum HY: Image super-resolution using gradient profile prior. In
*CVPR’08*. Anchorage, AK; 2008:1-8.Google Scholar - Bhat P, Curless B, Cohen M, Zitnick CL: Fourier analysis of the 2D screened poisson equation for gradient domain problems.
*Lecture Notes Comput. Sci*2008, 5303: 114-128. 10.1007/978-3-540-88688-4_9View ArticleGoogle Scholar - Saad Y:
*Iterative Methods for Sparse Linear Systems*. Philadelphia: SIAM; 2003.View ArticleGoogle Scholar - Davis TA:
*Direct Methods for Sparse Linear Systems*. Philadelphia: SIAM; 2006.View ArticleGoogle Scholar - Benzi M: Preconditionng techniques for large linear systems: a survey.
*J. Comput. Phys*2002, 182(2):418-477. 10.1006/jcph.2002.7176MathSciNetView ArticleGoogle Scholar - FFTW [http://www.fftw.org]
- Intel Integrated Performance Premitive [http://software.intel.com/en-us/articles/intel-ipp/]
- Li N, Saad Y, Chow Y: Crout versions of ILU for general sparse matrices.
*SIAM J. Sci. Comput*2003, 25(2):716-728. 10.1137/S1064827502405094MathSciNetView ArticleGoogle Scholar - Mertens T, Kautz J, Van Reeth F: Exposure fusion. In
*Pacific Graphics*. Maui, HI; 2007:382-390.Google Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.