Bilateral image denoising in the Laplacian subbands

Abstract

This paper presents an image denoising algorithm, which applies bilateral filtering (BLF) in the Laplacian subbands. It is noted that the subband images have wider area of photometric similarity than the original, and hence, they can be more benefited by the BLF than the original. Specifically, an image is Gaussian filtered to obtain a low band image, and the low band image is subtracted from the original to have the high band signal, which forms the Laplacian subbands. For the high band image denoising, we derive an adaptive kernel that is dependent on the edge intensity and photometric similarity of subband images. The high band image is convolved with this kernel and then added to the denoised low band signal, which produces the denoised image. We also propose to process the denoised high band signal by the gradient histogram preservation method, for sharpening the edges with less noise amplification. Experimental results show that the proposed denoising method provides higher PSNR than the original BLF and other multi-resolution denoising algorithms. Since the high band image is also effectively denoised in this process, the sharpened image by high band modification is also visually more pleasing when compared with the results of the conventional sharpening methods.

1 Introduction

Image denoising is a fundamental process in image formation, transmission and display systems, and thus a huge number of methods have been developed. The overview of classical linear filtering and some of recently developed nonlinear methods can be found in [1], where the relations of different nonlinear methods are also well explained. For suppressing the noise while keeping the edges, the state-of-the-art methods use the similarities of pixels locally or globally. For example, a simple yet effective local-similarity method is the bilateral filtering (BLF) [2, 3], and the representatives of global similarity methods are nonlocal means (NLM) filtering [4] and block matching 3D (BM3D) algorithm [5].

The edge-sharpening is also an important topic in image processing, which enhances the visual quality of images [6–9]. One of the classical edge enhancement methods is to use the “unsharpening filter”, where an image is low pass filtered and subtracted from the original, which leaves the high band signal that contains the edges. The high band signal is then amplified and added to the low pass filtered image, which is the edge-sharpened result. When the Gaussian filter is used for the low pass filtering, its subtraction from the original is the Laplacian of Gaussian, and thus the subband images so formed are called the Laplacian Pyramids [6]. In this process, since the noise in the high band can also be amplified, it is necessary to denoise all the subband images in the Laplacian pyramid.

In this paper, we modify the BLF for denoising the Laplacian subband images, which is aimed as a new denoising algorithm that works better than the original BLF and also as an efficient method of suppressing high band noise when sharpening the edges in the Laplacian pyramids. The idea of applying the BLF to the Laplacian subband images is based on the observation that the BLF works better when there are more photometric similarities in the images, and the subband images have wider area of photometric similarity than the original. However, since the properties of subband images are different from the original, we need to design a new filtering kernel, which is one of the modifications proposed in this paper. Also, for the edge enhancement with noise suppression, we propose a new enhancement technique which restores the strength of edges that are smoothed by filtering and then adds the restored edges to the high band signal.

Experiments on the images corrupted by pseudo white Gaussian noise, shot noise, and mixture noise are performed, and it is shown that the proposed method improves PSNR than the original BLF and other local neighbor methods based on the subband decomposition. For evaluating the results for the real camera noises, we capture noisy images under low light conditions and compare the visual qualities. Also, considering that the addition of multiple tripod captured images as a reference, we compare the PSNR. Comparisons on real noisy images also show that the BLF in the Laplacian subbands improves the denoising performance than the original BLF. Also, modification of the edge coefficients in the high band gives sharpened images with less noise amplification than the conventional edge-sharpening method in the Laplacian pyramids. When compared with the nonlocal approaches such as NLM [4] and BM3D [5], the proposed method shows lower or similar PSNR for the white Gaussian noise like any other local adaptive filters. However, for the real noise and mixed noise, the proposed method shows comparable or sometimes higher PSNR than the nonlocal methods while requiring much less computations due to the nature of local filtering. In summary, the proposed method shows better results than the conventional BLF and other subband filtering schemes such as [10, 11], which are the representatives of local adaptive filtering methods, and shows comparable results to the nonlocal methods for the nonstationary noises while requiring less computations. Hence, the Laplacian subband BLF can be a reasonable choice for denoising and enhancing the images when fast or real-time implementation is needed.

2 Related works

2.1 Laplacian subbands

When a signal is low pass filtered and the filtered output is subtracted from the original, then we obtain a high band signal. If this process is repeated for the low pass filtered signal, then we obtain a set of subband signals. The Laplacian pyramid for an image is constructed in this manner, where the low pass filter is a Gaussian filter with appropriate kernel bandwidth. More specifically, for a given image I, the Gaussian filter is applied iteratively with downsampling at every step. This process can be described as

$$\begin{array}{*{20}l} G_{0} &= I, \\ G_{k+1} &= \downarrow_{2}(Gaussian(G_{k})) \quad for \quad k=0,\ldots,n-1 \end{array} $$

((1))

where ↓ ₂(·) denotes the downsampling by 2 and G a u s s i a n(·) is the Gaussian filtering. Then, the Laplacian subbands are defined as

$$\begin{array}{*{20}l} L_{k+1} &= G_{k} - \uparrow_{2}(G_{k+1}) \qquad for \quad k=0,\ldots,n-1 \\ L_{n+1} &=G_{n} \end{array} $$

((2))

where ↑ ₂(·) denotes the upsampling by 2 and n is the level of pyramid. In this paper, we use just two levels of Laplacian subband (n=1), where L ₁ denotes the high-frequency subband and L ₂=G ₁ represents the low-frequency subband.

2.2 Bilateral filter

The bilateral filter is a nonlinear filter that considers both of spatial and photometric distances of neighboring pixels. Specifically, the filter output is defined as

$$ J(p) = \frac{1}{W}\sum_{q \in N_{p}} w(p,q) I(q) $$

((3))

where p and q denote pixel positions, N _p is the neighbor of p, I(q) is the intensity of input image at a pixel q, W is the normalizing factor $W=\sum _{q \in N_{p}} w(p,q),$ and w(p,q) is the kernel of the BLF defined as [2]

$$ w(p,q) = \exp \left(-\frac{\|p-q\|^{2}}{2{\sigma_{d}^{2}}}\right) \exp \left(-\frac{\|I(p)-I(q)\|^{2}}{2{\sigma_{r}^{2}}}\right) $$

((4))

where σ _d is the bandwidth for the spatial distance and σ _r for the photometric distance. For successfully reducing noise variance while keeping the edges, it is important to find the balance between σ _d and σ _r , and also to find an appropriate size of the neighbor.

3 Bilateral filtering in the Laplacian subbands

3.1 Example of subband BLF for a 1-D signal

We first show a simple denoising example with a synthetic 1-D signal, which motivates to apply the BLF to Laplacian subbands. Note that the kernel of bilateral filter in Eq. (4) is consisted of two terms, i.e., geometric and photometric terms. From this, we can see that the photometric weights would be kept large for wider area when a pixel p is in the flat area where I(p) and I(q) are similar, and hence many neighboring pixels can contribute for the denoising. On the contrary, when the pixel is in the non-flat area where ∥I(p)−I(q)∥ is large, the photometric weights diminish and thus the neighboring pixels less contribute for the denoising.

Figure 1 is the illustration for our example, where Fig. 1 a is the original noisy signal, and Fig. 1 b, c show its Laplacian low and high subbands. It can be seen that the slope area in the original signal becomes a flat area in the high band, and thus more noise reduction can be gained here due to the widened area of photometric similarity. For these three signals, we apply the BLF with the same parameters, ${\sigma _{d}^{2}}=4$ and ${\sigma _{r}^{2}}=49$, and the denoising results of overall area and specific areas (flat, edge, and slope areas as in Fig. 1 a) are summarized in Table 1. In the table, “original” column is the mean square error (MSE) of a noisy signal, “original BLF” represents the BLF result of this signal, and “subband BLF” means to apply the BLF to both of low and high band signals in Fig. 1 b, c and then add them. Also, the “overall” means the MSE of the overall region of the signal in [0,255], and flat, edge and slope represent the areas as defined in Fig. 1 a. At the first row, it can be seen that the BLF of the signal greatly reduces the noise variance (from 9.12 to 3.24) and the subband BLF reduces the variance further (to 2.11). When comparing the area-wise results, it can be observed that the ratio of the denoising gain is the most significant in the slope area, whereas the subband BLF has worse gain at the edge area. Hence, this supports our motivation that the subband BLF has the possibility of increasing the denoising performance at the slope areas and possibly slowly varying areas which become flat in the high band. From the toy example explained above, it can be seen that the main challenge in applying the BLF to the subbands is to find a method to mitigate the noise in the edge area and also to find appropriate filtering parameters.

Fig. 1

An example of Laplacian subband decomposition for a 1-D signal. a Original signal. b Low band signal. c High band signal

Table 1 Mean squared errors of the BLF results for the original and subband signals

3.2 Proposed subband BLF

For a given input image, we first perform subband decomposition as Eqs. (1), (2) to obtain the low band signal L ₂ and high band L ₁. For the low band image L ₂, we apply the conventional BLF with σ _d =1.8 and σ _r =σ as suggested in [10], where σ is the noise variance. As stated above, we concentrate on the filtering scheme for the high band image L ₁, especially at the edge area. The basic idea is to give larger weights to the pixels that have similar edge intensities as well as pixel intensities. Also, when it is highly probable that a pixel is on the edge, it needs to be less affected by the neighboring pixels. These ideas are encoded into a new guidance term in addition to Eq.(4) as

$$ \begin{aligned} w(p,q) = &\exp \left(-\frac{\|p-q\|^{2}}{2{\sigma_{d}^{2}}}\right) \exp \left(-\frac{\|I(p)-I(q)\|^{2}}{2{\sigma_{r}^{2}}}\right)\\ &\exp \left(-\frac{\|h(p)-h(q)\|^{2}}{2{\sigma_{h}^{2}}(p)}\right) \end{aligned} $$

((5))

where ${\sigma _{h}^{2}}(p)$ is the pixel dependent bandwidth, and h(p) is the intensity of the pixel p in the histogram-equalized image of L ₁ which will be explained later in more detail. Comparing this kernel with that of the original BLF in Eq. (4), the third term is our proposal which adaptively controls the weights near the edge areas. The adaptive bandwidth for the BLF has already been considered in [9], where the σ _r is adjusted along with an offset parameter by the optimization method with some training images. Unlike this previous adaptive BLF, our method is quite a simple algorithm which adjusts σ _h in the new kernel depending on whether the pixel is on the edge or not.

In summary, our method employs a new guidance image h(p) in the manner of joint bilateral filtering [3], for reducing the weights on the edge pixels and vice versa. For this, we let the bandwidth in the new term to be pixel dependent, i.e., the pixel difference in the high band (∥h(p)−h(q)∥) is considered in weight control. Precisely, our method adjusts σ _h (p) to $2\sqrt {2}\sigma$ or $4\sqrt {2}\sigma$ depending on the edge strength of given image.

For these edge-dependent modifications, we extract edge information from the BLF of low band image L ₂, which is denoted as $\hat {L_{2}}$. For determining whether a pixel is an edge pixel or not, we apply the Laplacian of Gaussian filter and then thresholding. Specifically, we convolve $\hat {L_{2}}$ with the kernel defined as

$$ LoG(x,y) = -\frac{1}{\pi\sigma^{4}} \left(\!\!1-\frac{x^{2}+y^{2}}{2\sigma^{2}}\!\right) \exp\left(\!\!-\frac{x^{2}+y^{2}}{2\sigma^{2}}\!\right) $$

((6))

and then the output pixels larger than 75 % of the mean value are considered the edge pixels. This gives an edge map E(p) which is 1 when the pixel p belongs to edge pixels, and 0 if not. For simplicity, the edge map is obtained from the approximate intensity component (image of (R+G+B)/3), and this edge map is applied to all of color components equally. With this edge map, the kernel bandwidth is determined as

$$ \sigma_{h}(p)= \left\{ \begin{array}{ll} 2\sqrt{2}\sigma & \textrm{if \(E(p)=1\)}\\ 4\sqrt{2}\sigma & \textrm{otherwise.} \end{array} \right. $$

((7))

It can be seen that the kernel bandwidth is small when the pixel is on the edge, so that the neighboring pixels less contribute to the averaging and thus the edge intensities are less changed. Conversely, the pixels in the flat areas are more strongly filtered than the edge pixels. It is worth to mention that we use ∥h(p)−h(q)∥ (histogram equalized intensities of L ₁ into the range [0,255]) instead of ∥L ₁(p)−L ₁(q)∥, because L ₁(p) can have negative value and its dynamic range is large. Denoting the output of proposed BLF of L ₁ as $\hat {L}_{1}$, the final denoised image is obtained as $\hat {L}_{1} + \hat {L}_{2}$.

Throughout the experiments, it is found that the low-band (L ₂) filtering with a variety of parameter changes does not much affect the overall performance. Hence, we apply just the original BLF with σ _r =σ for the low-band filtering, and we have focused on the kernel design for the high-frequency subband L ₁. Also, when comparing the results between the adaptive bandwidth and non-adaptive bandwidth (when σ _h is fixed), the gain by the adaptive scheme is not significant (under 0.1dB PSNR gain) because the edge area is small compared to others.

The overall process is illustrated in Fig. 2, where λ=1 corresponds to the proposed Laplacian subband filtering explained above, and 0≤λ<1 gives the edge enhanced results to be explained in the next section.

Fig. 2

Block diagram of proposed BLF (λ=1) and edge enhancement method. (0≤λ<1)

3.3 Image enhancement with the Laplacian subband denoising

As stated in the introduction, edge enhancement can also be easily achieved by manipulating the denoised high band signal in the Laplacian subbands. A straightforward method would be to amplify the denoised high band and then add this signal to the denoised low band image, like the original unsharp mask method. However, since the edge components in the high band have been smoothed in the filtering process, the straightforward method might add the smeared edges. Hence, we try to restore the edge strength of the high band image as strong as the original one, and then add this restored edges. For this, we adopt the idea of gradient histogram preservation (GHP) in [12], which is to impose a constraint that the processed image has the same gradient histogram as the estimated original one. Specifically, for the noisy image model:

$$ {\mathbf y}={\mathbf x}+{\mathbf v} $$

((8))

where x is the original image, v is the noise, and y is the observed noisy image, the processed image is constrained to have similar gradient histogram as x. In [12], considering the histogram of gradients of y as the discretization of the pdf of gradient distribution of y, the gradient histogram of the original image x is found by solving

$$ \arg\min_{H_{x}}\{\|H_{y} - H_{x} \otimes H_{v}\|^{2}+c\cdot R(H_{x})\} $$

((9))

where H _x , H _y , and H _v are the gradient histograms of x, y, and v respectively, ⊗ is the convolution operator, and c·R(H _x ) is a regularization term. For solving this problem, H _y is estimated from the observed data and H _v is modeled as a hyper-Laplacian distribution as [12]:

$$ p_{x} = k\exp(-\kappa|x|^{\gamma}) $$

((10))

where k is normalization factor.

Note that the high band image L ₁ in our subband BLF scheme is also a kind of gradient image, where the above GHP approach can be applied. Applying the Laplacian subband decomposition to Eq. (8), we have the high band relationship as

$$ \mathcal{L}_{1}({\mathbf y})=\mathcal{L}_{1}({\mathbf x})+\mathcal{L}_{1}({\mathbf v}) $$

((11))

where $\mathcal {L}_{1}(\cdot)$ is the operator that extracts high band signal of the input image, and hence $\mathcal {L}_{1}({\mathbf y}) = L_{1}$ in our problem. Like the GHP approach, we wish to find the histogram of $\mathcal {L}_{1}({\mathbf x})$ so that we match the histogram of $\hat {L_{1}}$ to this one. Denoting the histogram of $\mathcal {L}_{1}({\mathbf x})$ as the “reference histogram” H _r , we obtain it in a similar manner as Eq. (9), except that the positive and negative coefficients are considered separately in order not to diminish the peaks of coefficients that appear around the edges. To be specific, we obtain H _r as

$$ H_{r}\,=\,\arg\min_{H}\{\!\|H_{y,+}-\!H\otimes H_{v}\!\|^{2}+\!\|\!H_{y,-} +H\otimes H_{v}\|^{2}+c\cdot R(H_{r})\}. $$

((12))

where H _y,+ is the histogram of positive values in $\hat {L_{1}}$, H _y,− for the negative values, and H _v is the histogram of $\mathcal {L}_{1}({\mathbf v})$ that is modeled as Eq. (10). The range of parameters for solving this problem is set the same as [12], i.e., κ∈[0.001,3] and γ∈[0.02,1.5]. Then, the histogram of L ₁ is matched to H _r , which is denoted as $\hat {L}_{1,matched}$ in Fig. 2 and the edge enhanced image is obtained as $\hat {L}_{1}=\lambda \cdot \hat {L}_{1,f}+\\(1-\lambda)\cdot \hat {L}_{1,matched}$.

4 Experimental results

4.1 Experiments on pseudo white and Poisson noise

To evaluate the performance of proposed method, images in Fig. 3 are used. Each image is corrupted by the additive white Gaussian noise (AWGN) with variance σ or Poisson noise with parameter Q. We compare our subband BLF (SBLF) algorithm with the original BLF [2], multiresolution bilateral filter (MBLF) [10], BLS-GSM [11], NLM [4], and BM3D [5] with the authors’ source codes. According to [10], we set σ _d =1.8 and σ _r =2σ. Also, 9×9 windows are used for the original BLF, MBLF, and the proposed method. The multiresolution BLF is implemented in MATLAB, and others are implemented in MATLAB and C/C++ through MATLAB MEX functions, and the codes are run on a PC with an Intel Core i5 CPU and 4 GB RAM.

Fig. 3

Test images from Kodak dataset, BSDS 500[13], etc

Table 2 shows the averaged denoising results for the images in Fig. 3, each of which are corrupted by 100 different pseudo random sequences of Gaussian distribution with σ=20,30,40,50. PSNR for each of the images and other experimental results are available at http://ispl.snu.ac.kr/~idealgod/SBLF, where our source code and full-resolution images of all the figures in this paper are also available. As shown in Table 2, the proposed method yields better results than BLF and MBLF, and comparable results with the BLS-GSM. When compared with the nonlocal methods, the proposed method shows better PSNR than NLM, but lower than BM3D. However, as shown in the last row of Table 2, the proposed method needs much less computation time than the nonlocal methods, as well as other local methods except the original BLF. Figure 4 is a sample set of restored images, which shows that the proposed SBLF provides better visual quality than other local methods, and comparable quality with BM3D.

Fig. 4

Results of denoising the images corrupted by AWGN with σ=40. a Original image. b Noisy image. c BLF, 26.9332 dB. d MBLF, 26.6292 dB. e BLS-GSM, 29.4673 dB. f NLM 29.2290 dB. g BM3D 30.2165 dB. h Proposed SBLF (λ=1 in Fig. 2), 29.0490 dB. i Proposed SBLF with high band enhancement (λ=0.3), 27.3006 dB

Table 2 Averaged PSNRs for AWGN

Table 3 shows denoising results when the images are corrupted by Poisson noise with Q∈{5,10,15}. It can be seen that the results show similar trends as the Gaussian noise case.

Table 3 Averaged PSNRs for Poisson noise

4.2 Experiments on mixed noise

Impulsive and mixed noises are also considered for evaluating the algorithm. In the experiments, images are corrupted by the mixture of impulsive noise with Poisson or Gaussian noise. Table 4 shows the PSNRs of denoised images which were corrupted by 20 % impulse noise. Table 5 shows the results for 20 % impulse noise + Gaussian noise with σ=10, and Table 6 presents the results for 10 % impulse noise + Poisson noise with Q=10. It can be seen that the proposed method shows comparable or sometimes better PSNR than the BM3D. The reason for these results seems that there are not much similar patches for the nonlocal methods when there are randomly distributed impulsive noises.

Table 4 PSNRs for mixed noise (20 % impulse noise)

Table 5 PSNRs for mixed noise (20 % impulse noise + Gaussian noise σ=10)

Table 6 PSNRs for mixed noise (10 % impulse noise + Poisson noise Q=10)

It is worth to note that all the algorithms need (estimated) noise variance as the input, for controlling the filter parameters. In the case of above simulated noises, we know the noise variance and use it for the kernel parameters. However, in the case of mixed noise and the real noise in the following subsection, we cannot know whether the estimate noise variance (by any of estimation methods) is accurate or not. Hence, we try many experiments with the input variance in the range of [10,70] and choose the best one for each of the algorithms.

4.3 Experiments with real noise

When an image is captured in an insufficient light condition, there appears noticeably strong noise. For the experiments on this kind of “real noise” (not the simulated noises as above), we test the algorithms on the images of indoor scenes. Since there is no ground truth image for the objective comparison in this case, we capture the scene 30 times with tripod and consider the average of these images as the reference image to compute PSNR. Table 7 shows PSNRs for several output images, and Figs. 5 and 6 show the images for subjective comparison. From the objective and subjective comparisons, it can be seen that the proposed method shows better results than the other local self-similarity methods, and comparable quality as the nonlocal similarity methods.

Fig. 5

Denoised results of real image 1. a Noisy image. b Reference image (average of 30 noisy images). c BLF, 36.6848 dB. d MBLF 35.5875 dB. e BLS-GSM 37.0479. f NLM, 37.8776 dB. g BM3D, 38.4951 dB. h Proposed SBLF (λ=1) 37.3870 dB. i SBLF (λ=0.3), 36.4595 dB

Fig. 6

Denoised results of real image 2. a Noisy image. b BLF. c MBLF. d BLS-GSM. e NLM. f BM3D. g SBLF (λ=1). h SBLF (λ=0.7)

Table 7 PSNRs for real noise

4.4 Noisy image enhancement

Finally, we present the results of the proposed image enhancement scheme, specifically the overall scheme of Fig. 2 with λ=0.3. For enhancing the noisy images, a plausible method would be to denoise the image first and then apply the conventional image enhancement methods. Since the proposed method is based on the BLF, the comparison is performed with the schemes that apply BLF first and then enhance the image with [6] or [8]. Figures 7 and 8 show these comparisons, where (a) is the original image, (b) is the noisy one, (c) is the result of sequentially applying BLF denoising and high band amplification, (d) is the BLF followed by edge aware local Laplacian filtering [6], (e) is the result of sequentially applying BLF denoising and guided filtering [8], (f) is the result of denoising via TEID [12], (g) is the result of ABF [9], and (h) is the output of proposed algorithm. The figures show that the proposed method effectively suppresses the noise while enhancing the texture and edges. In the case of [12] (Fig. 7 f and Fig. 8 f), it can be seen that the noise is well removed while “preserving” the textures. On the other hand, the results in Fig. 7 h and Fig. 8 h show that the proposed method enhances the texture area (especially feather areas and patterns around the eyes), because the proposed scheme with λ<1 adds the matched high frequency components to the denoised high band.

Fig. 7

Enhancement of noisy image (σ=20). a Original image. b Noisy image. c Result of sequentially applying BLF and high band amplification. d Sequentially applying BLF and edge aware local Laplacian filtering [6]. e Sequentially applying BLF and guided filtering [8]. f Texture enhanced imaged denoising (TEID) [12]. g ABF [9]. h Proposed SBLF (λ=0.3)

Fig. 8

Enhancement of noisy image (σ=20). a Original image. b Noisy image. c Result of sequentially applying BLF and high band amplification. d Sequentially applying BLF and edge aware local Laplacian filtering [6]. e Sequentially applying BLF and guided filtering [8]. f Texture-enhanced imaged denoising (TEID) [12]. g ABF [9]. h Proposed SBLF (λ=0.3)

5 Conclusions

We have proposed an image denoising method based on the Laplacian subband decomposition and BLF. The input image is decomposed into two subbands by the Laplacian of Gaussian, and the BLF is applied to each of the subbands with appropriate filtering kernel and parameter. The experiments show that the proposed method increases PSNR compared to the original BLF and other multi-resolution filtering methods. For the real noisy images, the proposed method also yields comparable results to the non-local similarity methods such as BM3D and NLM, while requiring less computation time. Since the proposed method is based on the Laplacian decomposition, the edge enhancement can also be efficiently achieved along with the denoising.