Weighted gradient domain image processing problems and their iterative solutions
© Kuk and Cho; licensee Springer. 2013
Received: 7 June 2012
Accepted: 22 November 2012
Published: 23 January 2013
This article explores an energy function and its minimization for the weighted gradient domain image processing, where variable weights are applied to the data term of conventional function for attaining better results in some applications. To be specific, larger weights are given to the regions where original pixel values need to be kept unchanged, like strong edge regions in the case of image sharpening application or high contrast regions when fusing multi-exposure images. In the literatures, it is shown that the solution to a constant weight problem can be efficiently obtained in the frequency domain without iterations, whereas the function with the varying weights can be minimized by solving a large sparse linear equation or by iterative methods such as conjugate gradient or preconditioned conjugate gradient (PCG) methods. In addition to introducing weighted gradient domain image processing problems, we also proposed a new approach to finding an efficient preconditioning matrix for this problem, which greatly reduces the condition number of the system matrix and thus reduces the number of iterations for the PCG process to reach the solution. We show that the system matrix for the constant weight problem is an appropriate preconditioner, in the sense that a sub-problem in the PCG is efficiently solved by the FFT and also it ensures the convergent splitting of the system matrix. For the simulation and experiments on some applications, it is shown that the proposed method requires less iteration, memory, and CPU time.
Since human visual system (HVS) is sensitive to the intensity changes, processing an image in the gradient domain often produces subjectively better results than the conventional intensity domain processing. Specifically, the gradient domain approach has been successfully applied to high dynamic range (HDR) imaging , image stitching [2–5], filtering , alignment , matching , etc. In the case of image stitching problems which include seamless cloning [2, 5] and image composite for panoramic view [3, 4], the gradient domain processing is considered the state-of-the-art method.
where d(x,y) is a data function which is usually the input image, and λ controls the balance between the terms. The data term has the role of keeping an output close to the input (in order not to be deviated from the input too much), while the gradient term matches image gradients to be close to the desired one. The analysis in  shows that the solution should satisfy screened Poisson equation which is well known in physics, and the equation can be solved directly (not iteratively) in the frequency domain. By using this formulation, a number of image processing applications can be handled such as sharpening, image stitching, and deblocking.
where the data weight w(x,y) ranges from 0 to 1. By controlling the weights depending on regional properties, we later show that a visually better result is obtained in some applications. The basic idea is to give larger weights to the pixels in noticeable regions where the edge is strong or contrast is high, so that well captured regions are kept intact and others are changed according to the manipulated gradients. Although the conventional iterative methods such as conjugate gradient (CG) or preconditioned conjugate gradient (PCG) methods can be used to obtain the solution of this general problem (varying weights) , it is noted that they require a large amount of memory and long computation time. Hence, we analyze the problem and derive a more efficient solver to this problem. In the analysis, we first show that the solution minimizing (3) should satisfy a variant of screened Poisson equation which we call inhomogeneously screened Poisson equation. Then, we solve the equation based on PCG where the system matrix of the screened Poisson equation is selected as a preconditioner. Since this solver obviates the need of explicit form of large linear system, it requires less memory space. We also show that this preconditioner implies the convergent splitting of system matrix. With the simulated and real data, we demonstrate that the proposed solver is faster than the PCG. Finally we show some examples of weighted gradient domain image processing, which provides better subjective/objective quality than the intensity domain solution and also the gradient domain solution without weights.
This article is organized as follows. The inhomogeneously screened Poisson equation is derived in Section 2.1 and a solution based on the PCG is presented in Section 2.2. The convergence rate is also analyzed in Section 2.3. In Section 3, the proposed formulation is applied to the image sharpening problem (Section 3.1) and exposure fusion problem (Section 3.2). Finally in Section 4, conclusions are given.
2 Variational formulation and its solution
In this section, we derive the inhomogeneously screened Poisson equation from the energy function in (3), which includes spatially varying data weight. Then we present an iterative solver for the equation and analyze its convergence rate.
2.1 Inhomogeneously screened Poisson equation
where i and j denote pixel indices, V is a set of all the pixels, N i is a set of 4 neighboring pixels of the i th pixel, and u i , d i , and g i j are discrete counterparts of u(x,y), d(x,y), and g(x,y), respectively.
2.2 Iterative solution
where W is a diagonal matrix with w i on the diagonal, b is a vector whose i th element is and L is a 5-point Laplacian matrix. It is noted that W−λ L in (8) is a large, sparse, and symmetric matrix, and it can be directly solved by triangular factorization. However, direct solver needs very large memory, even though the sparsity of the matrix is fully exploited . Hence, an iterative method is usually preferred to the direct method when solving this kind of problem, such as the preconditioned conjugate gradient (PCG). Preconditioning is to transform the original problem A u=b to the one that can converge faster, by pre-multiplying an inverse of a certain matrix that reduces the condition number (ratio of maximum to minimum eigenvalue in the case of symmetric matrix) of the resulting system. To be specific, the inverse of a preconditioning matrix M is multiplied to (8), i.e., M−1 A u=M−1 b, where A=W−λ L, so that the eigenvalue spread of M−1 A is smaller than that of original system matrix A. In summary, the PCG is to find a matrix M that reduces the condition number of the linear system, and then iterate the conjugate gradient steps as in the Algorithm 1.
Algorithm 1 PCG steps
u 0=0,r 0=b−A u 0, t 0=M−1 r 0, p 0=t 0
u i+1=u i +a p i
r i+1=r i -a A p i
t i+1=M-1 r i+1
p i+1=t i +ß p i
where σ(Θ) denotes the spectrum of λ, that is, the set of eigenvalues of λ.
where U−1 Γ U is a decomposition of the Laplacian matrix L and Γ is a diagonal matrix whose i th diagonal element γ i denotes the eigenvalue of Laplacian matrix. Since the Laplacian matrix is a negative semi-definite matrix (γ i ≤0 for all i), it follows that 1−λ γ i ≥1, where 1−λ γ i is the i th eigenvalue of A s . Hence, all the eigenvalues of A s is larger than 1. The other requirement for the convergence splitting, i.e., , can also be proved as follows.
Since both A and N are positive semi-definite and α≠1, , i.e., 0≤α<1. Hence . □
2.3 Convergence analysis
In this section, we analyze the convergence rate of the proposed preconditioner with the simulated data and real problems. We consider three measures for the evaluation of convergence rate, namely condition number, elapsed time and the amount of required memory. Specifically, the proposed method is compared with standard conjugate gradient without preconditioning, and several incomplete LU (ILU) factorization methods for the sparse symmetric system such as ILU0 , modified ILU (MILU)  and modified Crout variant of ILU (MILUC) .
Condition numbers for a p × p matrix
Required memory space (KB), elapsed time ( ms ) and required iterations to reach the error of 10 −3 for a 250000×250000 matrix
Required memory space (KB), elapsed tie ( ms ) and number of required iterations to reach the error of 10 −3 for the image sharpening example
Required memory space (KB), elapsed time (ms ) and number of required iterations to reach the error of 10 −3 for the HDR imaging problem
3 Examples of weighted gradient domain image processing
In this section, we present two examples, image sharpening and gradient domain exposure fusion, where the gradient domain processing with the variable weights can be more effective than the existing approaches.
3.1 Image sharpening
Image sharpening is one of the most commonly used techniques for enhancing the contrast. The conventional method for image sharpening is the Laplacian subtraction approach, i.e., a Gaussian blur is applied to a given image, the blurred image is subtracted from the original to form a contrast image, and then the contrast image is weighted and added to the original. In this procedure, since the subtraction of blurred image from the original can be interpreted as an approximation of Laplacian filtering, this technique is called Laplacian substraction.
where c 1(>1) and c 2 are constants, and s(x,y) is a vector image containing the saliency in both x and y directions around the pixel coordinate (x,y). According to the gradient manipulation strategy in (12), all the gradients are basically boosted by constant value c 1 and they are selectively magnified again by the edge saliency s(x,y). The parameter c 2 controls the amount of sharpening boosted by s(x,y). Although the gradient domain approach seems to be different from the Laplacian subtraction method, it has been shown that this approach can be interpreted as a generalized version of Laplacian subtraction method .
where g i is the gradient magnitude given by with d i v and d i h being the vertical and horizontal gradient value, respectively, and σ d controls attenuation. Note that the gradient manipulation term in our solution is designed in the same manner as in (12).
3.2 Weighted gradient domain exposure fusion
where ϕ(·) is a weight function which attenuates large gradients and magnifies small gradients, as defined in .
We have presented weighted gradient domain image processing problems where spatially varying weights are applied to the data term of conventional energy function. The problem is formulated as a system of linear equation, which is more efficiently solved by iterative methods such as PCG than by direct inverse or factorization methods. For solving the problem with the PCG, the most important thing is to find a preconditioning matrix that makes the system matrix have low condition number. We have shown that the system matrix for the constant weight problem is an appropriate choice for the preconditioner. Specifically, a subproblem in the PCG steps is efficiently solved by the proposed preconditioning matrix and it is also shown that this matrix induces the convergent splitting of the system matrix. Simulation and experiments also show that the proposed method converges faster than the existing methods, and also requires less memory space and CPU time. Applications of the weighted gradient domain image processing problems have also been presented, with the examples of image sharpening and the exposure fusion problems. In the comparison results, it is shown that better outputs are produced owing to the spatially varying design of data weight.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2012-0000913).
- Fattal R, Lischinski D, Michael W: Gradient domain high dynamic range compression. ACM Trans. Graph 2002, 21(3):249-256.View ArticleGoogle Scholar
- Pérez P, Gangnet M, Blake A: Poisson image editing. ACM Trans. Graph 2003, 22(3):313-318. 10.1145/882262.882269View ArticleGoogle Scholar
- Levin A, Zomet A, Peleg S, Weiss Y: Seamless image stitching in the gradient domain. Lecture Notes Comput. Sci 2004, 3024: 377-389. 10.1007/978-3-540-24673-2_31View ArticleGoogle Scholar
- Agarwala A: Efficient gradient-domain compositing using quadtrees. ACM Trans. Graph 2007, 26(3):94-98. 10.1145/1276377.1276495View ArticleGoogle Scholar
- Jia J, Sun J, Tang CK, Shum HY: Drag-and-drop Pasting. ACM Trans. Graph 2006, 25(3):631-637. 10.1145/1141911.1141934View ArticleGoogle Scholar
- Bhat P, Zitnick CL, Cohen M, Curless B: GradientShop: a gradient-domain optimization framework for image and video filtering. ACM Trans. Graph 2010, 29(2):1-14.View ArticleGoogle Scholar
- Baker S, Matthews I: Lucas-Kanade 20 years on: a unifying framework. Int. J. Comput. Vis 2004, 56(3):221-255.View ArticleGoogle Scholar
- Wu Y, Fan J: Contextual flow. In CVPR’09. Miami, FL; 2009:33-40.Google Scholar
- Sun J, Xu Z, Shum HY: Image super-resolution using gradient profile prior. In CVPR’08. Anchorage, AK; 2008:1-8.Google Scholar
- Bhat P, Curless B, Cohen M, Zitnick CL: Fourier analysis of the 2D screened poisson equation for gradient domain problems. Lecture Notes Comput. Sci 2008, 5303: 114-128. 10.1007/978-3-540-88688-4_9View ArticleGoogle Scholar
- Saad Y: Iterative Methods for Sparse Linear Systems. Philadelphia: SIAM; 2003.View ArticleGoogle Scholar
- Davis TA: Direct Methods for Sparse Linear Systems. Philadelphia: SIAM; 2006.View ArticleGoogle Scholar
- Benzi M: Preconditionng techniques for large linear systems: a survey. J. Comput. Phys 2002, 182(2):418-477. 10.1006/jcph.2002.7176MathSciNetView ArticleGoogle Scholar
- FFTW [http://www.fftw.org]
- Intel Integrated Performance Premitive [http://software.intel.com/en-us/articles/intel-ipp/]
- Li N, Saad Y, Chow Y: Crout versions of ILU for general sparse matrices. SIAM J. Sci. Comput 2003, 25(2):716-728. 10.1137/S1064827502405094MathSciNetView ArticleGoogle Scholar
- Mertens T, Kautz J, Van Reeth F: Exposure fusion. In Pacific Graphics. Maui, HI; 2007:382-390.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.