It is worth noting that the 3D transform in BM3D algorithm is separable. For example, we can first implement 2D transform on each block in every image block group, which is obtained by block matching, and then implement the 1D transform along the third dimension. Since the 2D transform is still local, there exist inevitable disadvantages of local transform, such as introduction of artifacts as well as blurry image edges after denoising. If an enough number of similar noisy image blocks can be obtained, the 1D transform can help denoise the noisy image very well. In particular, no transformation is required, as we can simply average those similar blocks in the case of added white Gaussian noise (AWGN) and achieve the ideal denoised results. Unfortunately, in reality, there are not many similar image blocks in the single image. Therefore, to avoid performing 2D transform on each image block, we can iteratively perform 1D transforms, thus better preserving edges and introducing less artifacts.
On the other hand, however, there are also two problems of iteratively performing 1D transforms. The first problem is that the image blocks are not completely the same. Even if we just use 1D transform, the image edges are only smoothed to a certain degree. The second problem is that some isolated strong noises will be retained if we just use 1D transform to denoise the image, since the transformed coefficient magnitude of strong noise will be very large, i.e., even larger than the coefficient magnitude of real signal. Thus, when we use a hard threshold to shrink the transformed coefficients, we cannot remove those strong noises. To solve these two problems and also achieve better image denoising results, we propose an improved BM3D algorithm. The respective improvements include (1) the use of adaptive block size, and (2) the use of multi-stage strategy. This improved BM3D algorithm includes four steps, as shown in Fig. 4 . As for the multi-stage strategy, we have presented more details in a previous conference paper [43].
Step 1: block matching 1D (Haar) transform domain filtering
In the original BM3D algorithm, the first step is block-matching 3D transform. Since the 2D transform on each block is still the local transform, some artifacts will be introduced after inverse transform. Also, since the transformed coefficient magnitude of strong noise is very large, the hard-thresholding operation can only remove the coefficients of those relatively small noises. To avoid the introduction of artifacts, we can only perform 1D transform along the third dimension, in which this 1D transform is a real non-local transform. So, in this step, we first implement block-matching operation according to the BM3D method, and then perform 1D Haar transform on the third dimension, i.e., the inter-block Haar transform. Note that we will not implement the 2D transform on each block, so we call this step as block matching 1D transform (BM1D). Also, since we just implement 1D transform, only a few noises can be separated from the real signal, while many other noises, especially strong noises, are still retained after this step. On the other hand, image edges can be better preserved and few artifacts are introduced.
To further efficiently preserve image edges, we use a perturbation strategy, i.e., we amplify the low frequency coefficients after the 1D transform as follows:
$$ {\widehat{B}}_G={T}^{-1}\left(\mathrm{shrink}\left(\left[{\left(T\left({B}_G\right)\right)}_1\bullet \gamma, {\left(T\left({B}_G\right)\right)}_2,\cdots, {\left(T\left({B}_G\right)\right)}_K\right]\right)\right) $$
(9)
where (T(BG))1 is the low frequency subband of the 1D transform on image block group BG, and γ > 1 is a gain factor, which is used to amplify the low frequency coefficients. T and T−1 are the forward and inverse 1D transforms, respectively. And shrink denotes a hard-thresholding operation. This perturbation strategy has two major functions: (1) it can protect low frequency coefficients without shrinkage, because most coefficients of image edges belong to low frequency subband after the 1D transform, and (2) it can reduce the effect of strong noise in the next step of block-matching, thus improving the block-matching accuracy. On the other hand, however, it will reduce denoising performance. So we next implement BM3D Wiener filtering to further denoise the result of this step.
Step 2: block matching 3D Wiener filtering
We use the resulted image of the first step as reference, and perform empirical Wiener filtering on the original noisy image. The purpose of using BM3D-based Wiener filtering in this step is to enhance weakened image edges after using the first step, as image edges are always weakened to some extent after applying the first step of noise reduction. The following is the empirical Wiener filtering:
$$ {\mathrm{Shrink}}_{\mathrm{wie}}\left(\theta \right)=\frac{\theta \bullet {\left|\widehat{\theta}\right|}^2}{{\left|\widehat{\theta}\right|}^2+{\sigma}^2} $$
(10)
where \( \widehat{\theta} \) is the block matching 3D transformed coefficients of the result image of the first step, θ is the block matching 3D transformed coefficients of the original noisy image, and σ is the standard deviation of noise. Actually, we can use a smaller σ in this step to better preserve image edges, the smaller σ can obtain the similar effects with the low frequency coefficients perturbation in the first step.
Note that here we use 3D transform, instead of 1D transform in the first step, because using 3D transform and hard-threshold operation can remove strong noises. Of course, 3D transform is not good at preserving image edges than 1D transform. After completing this second step, noises can be further removed partly.
Step 3: size-adaptive block matching 1D transform domain filtering
In this third step, we first perform DCT on each reference block before using the block matching operation, then use Eq. (6) to calculate the AC energy of the transformed coefficients, and finally determine the class of morphological component by Eq. (7), which will be used to adjust the block size according to the descriptions in Section 3. Note that this step is the most critical step in the whole algorithm. This is because, after the second step, there still exist a lot of noises. Moreover, since DCT is operated on each block of BM3D Wiener filtering in the second step, some strong noises become the certain pseudo textures, which are no longer subject to Gaussian distribution. Thus, we should not perform block matching on the results produced by the second step. To get rid of these noises with certain pseudo textures, we perform block matching on the original noisy image, and then use the block matching results to extract image blocks at the same locations to implement 1D Haar transform among blocks. Finally, we similarly use a hard-threshold operation to remove the noise, which can be effectively done except for few isolated strong noises.
It is worth noting that we use size-adaptive blocks to improve block-matching accuracy. In the smooth region, if we use a small size to perform block matching, the noise, especially isolated strong noises, would influence the block-matching result. In other words, it would result in noise matching, instead of real signal matching. In the contour region, if we use a large size to perform block matching, it would be difficult to obtain ideal contour matching result, as contours are illustrated as strait lines. However, if we use a small size to perform block matching in these contour regions, we can easily obtain accurate contour matching results. In the texture region, the block size is in-between the former two. This is because if the size is too small, the block matching result will be influenced by the noise; while, if the size is too big, it is still difficult to obtain accurate block matching result as the contour region.
Step 4: size-adaptive block matching 3D Wiener filtering
In the fourth step, similar to the third step, we first use Eqs. (5)–(7) to determine the class of morphological component for the reference block, and then adjust the size for the reference block. The rest of the procedure in this fourth step is the same as the classic BM3D Wiener filtering.
After completing the above four steps, we are able to obtain a final denoised image. Notably, all of the first three steps are equivalent to the first step in the original BM3D algorithm, i.e., the basic estimation stage in the original BM3D algorithm. Because we use both 1D and 3D transforms as well as the size adaptive block-matching in the proposed algorithm, we call the proposed algorithm as SA-BM1-3D in the rest of this paper. To better understand the procedure of SA-BM1-3D, we use Fig. 5 to show the results of each step.
To denoise color images, we also transform RGB image to luminance-chrominance image just like BM3D; there are three channels in a luminance-chrominance image, i.e., Y, U, and V, respectively. We also use the opponent color transformation to obtain luminance-chrominance image, its transform matrix refers to [2]. In each step of SA-BM1-3D, we use Y channel to perform block-matching, then apply the block-matching result to other two channels, i.e., U and V. We still use the proposed 4-step algorithm to denoise each channel.