- Research
- Open Access
On required accuracy of mixed noise parameter estimation for image enhancement via denoising
- Victoriya V Abramova^{1},
- Sergey K Abramov^{1},
- Vladimir V Lukin^{1}Email author,
- Karen O Egiazarian^{2} and
- Jaakko T Astola^{2}
https://doi.org/10.1186/1687-5281-2014-3
© Abramova et al.; licensee Springer. 2014
- Received: 29 March 2013
- Accepted: 23 December 2013
- Published: 14 January 2014
Abstract
Characteristics of noise (type, statistics, spatial correlation) are nowadays exploited in many image denoising and enhancement methods. However, these characteristics are often unknown, and they have to be extracted from an image at hand. There are many powerful and accurate blind methods for noise variance estimation for the cases of additive and multiplicative noise models. However, more complicated noise models containing a mixture of signal-independent (SI) and signal-dependent (SD) components are often more adequate in practice. Parameters of both components have to be automatically estimated to be used in image enhancement. This paper addresses a question of required accuracy of such estimation. Analysis is carried out for color images processed by a filter based on discrete cosine transform. The influence of errors in mixed noise parameters estimation is studied in terms of filtering efficiency. This efficiency is characterized by the conventional criterion peak signal-to-noise ratio (PSNR) and two visual quality metrics, PSNR human visual system masking (PSNR-HVS-M) and multi-scale structural similarity (MSSIM). If a reduction of filtering efficiency exceeds 0.5 dB (in terms of PSNR and PSNR-HVS-M) or 0.005 (in terms of MSSIM), mixed noise parameters estimation is assumed to be unacceptable. As the result, it is shown that SI and SD noise parameters have to be estimated with a relative error not exceeding 20%…30%.
Keywords
- Mixed noise
- Blind variance estimation
- Filtering efficiency
- Visual quality
1. Introduction
Imaging systems (sensors) are widely used in various applications such as remote sensing, non-destructive control, medical diagnostics, photography, etc. [1]. Some acquired images need a special pre-processing (e.g., denoising, deblurring, edge detection, compression [2–4], etc.) for further exploitation (e.g., object recognition, visual inspection, diagnostics, etc.). In order to enhance an image using, e.g., modern image denoising methods based on wavelets [5–7] or discrete cosine transform (DCT) [8, 9] transforms, one has to know a noise type and its basic characteristics such as probability density function (PDF), variance, or two-dimensional (2D) spatial correlation function (if the observed noise is not independent and identically distributed (i.i.d.)). Proper thresholds in edge detection and image segmentation depend on noise statistics as well [1, 10]. In lossy image compression, a quantization step has to be adaptively adjusted depending on noise variance [11].
A practical problem is that a priori information on noise type and basic characteristics is not always available. Although there are such applications as synthetic aperture radar (SAR) imaging with known number of looks and image forming mode for which speckle characteristics can be accurately predicted [10], it appears a more practical situation when noise characteristics are fully or partly unknown (unavailable). For example, for color images acquired by digital cameras, noise properties are determined by camera settings, illumination conditions, and other factors [9, 11–13]. Similarly, noise characteristics might be considerably different in sub-band images of multi- and hyperspectral remote sensing data acquired from airborne and space-borne platforms [14–16].
Then, a task of blind estimation of noise characteristics for each particular image subject to further processing, in particular, denoising [9, 11, 12] becomes of prime importance. General requirements to the methods intended for blind (automatic) estimation of additive and multiplicative noise variance can be found in [8, 17]. Clearly, it is desirable to provide almost unbiased estimates with estimation variance as small as possible. It is also needed to ensure applicability of a method to images of different content (including highly textural ones) and different noise levels (intensities) including non-intensive noise. A blind estimation method is practical if it is fast enough.
Moreover, for the methods of pure additive or multiplicative noise variance estimation, it has been established that the estimation relative error in practice should not be larger than ±20% [8, 18]. If an obtained variance estimate is outside this limit, under- or oversmoothing takes place in image denoising based on blind estimation result.
With a new generation of sensors, signal-dependent noise models have become more popular since they describe noise statistics better [8, 9, 11–13, 15, 16, 19, 20]. There is an essential interest in design and testing the methods for blind estimation of mixed noise parameters [9, 12, 15, 16, 21, 22]. However, to our best knowledge, it has not been studied yet how accurate have to be these estimates. It is worth mentioning that dependence of noise variance on signal (local mean in images) can be set parametrically, i.e., by some polynomial [9, 12, 15, 22]. To avoid difficulties of polynomial order choice, we consider below a simplest case of mixed noise where SD and SI components are characterized by one parameter each. This model is typical in raw data in digital photos [9, 11], sub-band images of hyperspectral remote sensing data [15, 16] and radar images formed by multi-look SARs and side-look aperture radars [21, 22].
Being applied at the first stage of image processing chain [8, 9, 11, 12], blind estimation of noise parameters provides data for next stages. Therefore, accuracy of blind estimation has to be analyzed in the combination with efficiency of further image processing. Since image filtering is one of the most common operations (stages) of image processing chain, we have decided to carry out our analysis just from the viewpoint of filtering efficiency. We consider estimation accuracy acceptable if estimation errors that always happen in practice do not produce essential reduction of image denoising efficiency. In the next sections of the paper, we give quantitative definition of what is essential. We characterize image filtering efficiency not only by the conventional criteria, such as peak signal-to-noise ratio (PSNR) (or mean square error (MSE)), but also using metrics that describe image visual quality [23]. This is important since for many considered applications, visual quality of filtered images has a great importance (e.g., digital photography, medical imaging). The main goal of this paper is to give practical recommendations (requirements) on accuracy of mixed noise parameters estimation to ensure efficient image filtering.
2. Noise models and estimated parameters
In general, noise is considered signal-dependent if its statistical characteristics (variance, PDF) depend upon information signal (image). A typical example of signal-dependent noise is Poisson noise for which variance is equal to the true value of image pixel ${\mathrm{\sigma}}_{\mathrm{sd}}^{2}={\mathrm{I}}^{\mathrm{tr}}$ and noise PDF changes. It is close to Gaussian for large true values but considerably differs from Gaussian if I^{tr} is small (less than 12). Other examples of signal-dependent noise are film-grain noise and speckle [24, 25]. For all these cases, a dependence of signal-dependent noise variance on true value ${\mathrm{\sigma}}_{\mathrm{sd}}^{2}=f\left({\mathrm{I}}^{\mathrm{tr}}\right)$ is monotonically increasing (although other character of this dependence is, in general, possible).
The aforementioned examples relate to the cases when there is a single source of noise. However, noise can originate from several different sources, for example, image pixel generation (photon counting, coherent processing of registered signals) and circuitry (thermal) noise [19]. Then, one deals with a mixed noise, such as mixed additive and impulse noise [26, 27] where impulse noise usually originates from coding/decoding errors at image transmission via communication channels. However, here, we address another type of mixed noise that has SI and SD components. A prominent example is noise in modern imaging sensors namely dark noise, thermal noise, and photon-counting noise [19].
where ${\mathrm{\sigma}}_{\mathrm{si}}^{2}$ is variance of SI noise, e.g., of dark current noise and k is a proportionality factor for SD component [9, 15]. The model is valid for raw images acquired by digital cameras and sub-band images formed by hyperspectral sensors.
for the model (2), where I_{IM}, J_{IM} define image size. If MSE_{inp} is twice larger than ${\mathrm{\sigma}}_{\mathrm{si}}^{2}$, the impact of SD noise can be considered prevailing (the SD noise is considered dominant) and vice versa. Note that MSE_{inp} can be also treated as equivalent variance of the noise in original images.
Such preliminary analysis can be useful since both situations can be met in practice. For example, SI noise is usually assumed to be dominant for sub-band images formed by old generation sensors, e.g., AVIRIS [14, 15] while SI is prevailing for new generation sensors [15, 16]. Thus, it is desirable to study both situations in our further analysis.
It is also supposed that noise is spatially uncorrelated. Although this is often not true in practice, this assumption simplifies our analysis. In the future, we plan to carry out similar analysis for spatially correlated noise as well.
3. Denoising method and quantitative criteria
There is a great number of denoising methods, especially for processing color images (note that just color image case for which enhancement is of great value will be analyzed below) [27, 29, 30]. However, here, we are not interested in image denoising methods that do not exploit information on noise type and statistics in their operation. Instead, we have to focus on filtering methods able to adapt to quite complex types of signal-dependent noise [8, 9, 31] described by the models (1) and (2).
Then, there are two possible approaches. The first one is to apply a variance-stabilizing transformation [32, 33] that converts an image corrupted by a given type of signal-dependent noise to an image corrupted by an additive noise and then to apply a filter designed to suppress an additive noise. For the model (1), this can be done by, e.g., generalized Anscombe transformation [33].
The second possibility is to apply denoising methods that can be adapted to signal-dependent nature of the noise. Since the DCT-based filtering allows to do this easily [34, 35], we concentrate below on this method of denoising. The DCT-based filtering [34, 35] is carried out in blocks of a limited support, usually 8 × 8 pixels. This feature of the DCT-based filtering allows easy adaptation to non-stationary (signal-dependent) properties of the noise. Direct DCT is performed in each block. Then, a thresholding of DCT coefficients (hard, soft, or combined [36]) is performed for all coefficients except DC. Here, we focus on hard thresholding since it is simple and the most efficient in terms of provided output PSNR [36].
For signal-dependent noise, absolute values of DCT coefficients in an 8 × 8 block, D(n, m, q, s), have to be compared with a threshold $T\left(n,m\right)=\beta \sqrt{f\left({\overline{I}}_{\mathit{nm}}\right)}$, where indices n,m define a block upper left corner, q = 0,..,7, s = 0,…,7 are indices of DCT coefficients in 8 × 8 blocks, ${\overline{I}}_{\mathit{nm}}$ is nm-th block local mean, and β is a constant. Hard thresholding operation is resulting in assigning to zero those coefficients (D(n, m, q, s), q = 0,…,7, s = 0,…,7 (except DC component with q = 0 and s = 0) which absolute value is below the threshold: |D(n, m, q, s)| < T(n, m), and keeping all others unchanged (for each possible position of a block). After this, an inverse DCT is applied to the thresholded DCT coefficients in each block. Here, we consider the DCT-based filtering with full overlapping (shifting by one position to the next window), thus, multiple denoised (filtered) values are obtained for each image pixel (except those ones placed at the image corners). These multiple denoised values of the same pixel coming from different windows are averaged. This procedure is similar to a translation invariant wavelet shrinkage, where instead of block DCTs, wavelet transform of an image and all possible shifted version of it are performed. This allows to improve a denoising performance and to diminish blocking artifacts with respect to the case of filtering performed in non-overlapping blocks. Note also, that in the case of fully overlapping blocks, DCT-based filtering efficiency is close to that of the state-of-the-art filters [23] such as, e.g., BM3D [37]. This is one more reason why the DCT-based filtering was selected in our analysis.
Usually, the value of β recommended in hard thresholding is 2.6 [36] although one can find slightly different recommendations [38] which we will discuss later.
respectively. In practice, if ${\sigma}_{\mathrm{si}}^{2}$ and k are estimated, the obtained estimates of these parameters are to be used in (5) and (6).
A traditional approach to filter efficiency characterization and comparison to other filters is to calculate and analyze output MSE or PSNR. In this paper, we mainly consider R component of RGB color image. Results for other color components are in good agreement with those for the R component denoising (this will be shown by particular examples).
Since we deal with color images (and their components) that are usually intended for visual inspection, visual quality of processed images is of a value. Thus, alongside with the conventional output PSNR (PSNR = 10 log _{10}(255^{2}/MSE_{out}) where MSE_{out} denotes MSE after denoising), it is worth using visual quality metrics.
In our analysis, we have used two quality metrics inspired by human visual system (HVS). A first one is the recently proposed metric PSNR human visual system masking (PSNR-HVS-M) [39] (available at [40] and defined as PSNR-HVS-M = 10 log _{10}(255^{2}/MSE_{out}^{HVS}), where MSE_{out}^{HVS} is a specific MSE determined in DCT domain that takes into account such peculiarities of human vision system as different sensitivity to distortions in different spatial frequencies and masking effects). This metric is among the best in characterizing visual quality of images corrupted by a noise as well as images with distortions due to filtering and compression [41]. PSNR-HVS-M is intended to assess visual quality of both gray-scale and color images. Similarly to the conventional PSNR, the metric PSNR-HVS-M is expressed in decibels and larger values correspond to better visual quality. If PSNR-HVS-M exceeds 40 dB, distortions can be hardly noticed [42].
Another widely used HVS metric is multi-scale structural similarity (MSSIM) [43]. It takes into account human’s ability to adapt to a structural similarity at different scales and HVS sensitivity to distortions of luminance and contrast. The metric values vary from 0 (extremely bad quality) to 1 (perfect or ideal quality). As it is seen, this metric has another range of variation where the value 0.99 corresponds to practical invisibility of distortions if they are spread over entire image [42]. While PSNR-HVS-M exploits discrete cosine transform in blocks for its calculation, MSSIM is based on wavelets. Thus, we can expect that if analysis of both metrics leads to drawing similar conclusions, then these conclusions will be well grounded.
Note that performance analysis of filtering efficiency is usually carried out under an assumption that noise characteristics are known in advance (or accurately pre-estimated) and filter parameters are set according to certain recommendations [6, 9, 31]. Recall that, in our case, the thresholds for blocks are set to $T\left(n,m\right)=\beta \sqrt{{\widehat{\sigma}}_{\mathit{si}}^{2}+\widehat{k}{\overline{I}}_{\mathit{nm}}}$ or $T\left(n,m\right)=\beta \sqrt{{\widehat{\sigma}}_{\mathit{si}}^{2}+\widehat{k}{\overline{I}}_{\mathit{nm}}^{2}}$ depending on the considered model where ${\widehat{\sigma}}_{\mathit{si}}^{2}$ and $\widehat{k}$ are model parameter estimates obtained by some technique. Thus, all three parameters, β, ${\widehat{\sigma}}_{\mathit{si}}^{2}$ and $\widehat{k}$, can influence filter performance. Besides, image properties have also an impact on efficiency of filtering.
Assume that statistical characteristics of the noise are not known in advance. Properties of image signal component (for example, its spatial spectrum in DCT or wavelet domain) are also unknown since an observed image is noisy. Thus, to partly simplify the situation, we have, at least, to set the parameter β fixed. Let us demonstrate that setting β equal to 2.6 is a good choice for the considered hard threshold DCT filter with full overlapping of blocks.
The observed tendencies are the following. First, the average value of β is really about 2.6 for all three sets of the noise parameters. Second, there are complex structure images (e.g., the test images #1 and #13) for which optimal β are slightly smaller than 2.6. There are also simple structure images (#3, 7, 20, 23, 25) for which the optimal β can be slightly larger than 2.6, especially if noise is quite intensive (e.g., k = 1 and ${\sigma}_{\mathrm{si}}^{2}$ = 30).
Dependences for the metric MSSIM (Figure 2b) are very similar. The only difference is that optimal values of β are by about 3% smaller than for the corresponding cases in Figure 2a. This tendency has been earlier observed in [38]. It can be explained by the fact that by setting a slightly smaller β one provides better edge/detail/texture preservation while noise suppression in homogeneous image regions becomes worse. This is just the case when a filtered image is perceived as having better visual quality. Dependences of optimal β for another HVS metric, PSNR-HVS-M (not presented in the paper), are similar to those ones for PSNR. The difference is again in smaller optimal β similarly to MSSIM.
For more detailed analysis, we have further concentrated on two color images from the database TID2008 [41], namely the test image #3 (one of the simplest) and the test image #13 (the most complex one) (see Figure 1) since, according to our previous experience [17, 18], just these marginal cases determine basic requirements. The test image #25 has not been chosen since it is artificial and we are more interested in enhancing natural scene images. The test image #20 has not been used for the analysis since clipping (overexposure) effects occur for it in bright upper region that corresponds to sky.
Our idea is that a joint analysis of filtering efficiency for these two images carried out for different noise parameters’ sets can produce initial insight on basic requirements to an accuracy of blind estimation of noise parameters ${\sigma}_{\mathrm{si}}^{2}$ and k. We assume that these requirements will be correct for the majority of real-life images. At the end of the paper, we will check validity of these requirements for two extreme cases and for entire database of images corrupted by mixed noise with different sets of parameters.
where ${\widehat{\sigma}}_{\mathit{si}}^{2}$ and k are estimates of ${\sigma}_{\mathrm{si}}^{2}$ and k, respectively. Then, δ_{ V } = 0 and δ_{ k } = 0 correspond to true values of ${\sigma}_{\mathrm{si}}^{2}$ and k; δ_{ V } = - 1 relates to the case when the additive noise component is absent. Similarly, δ_{ k } = -1 corresponds to assumption that the signal-dependent noise component is absent and the present noise is pure additive. The case δ_{ V } = -1 and δ_{ k } = -1 relates to the case of no filtering applied, i.e., to a noisy image.
In our experiments, we have analyzed δ_{ V } and δ_{ k } ranging from -1 to +1 (from -100% to +100%), that is for δ_{ V } = 1 and/or δ_{ k } = 1, it is assumed that the estimates of ${\widehat{\sigma}}_{\mathit{si}}^{2}$ and $\widehat{k}$ are twice (by 100%) larger than the true values of these parameters. According to our experience [22], practical estimates are rarely outside these limits.
Thus, both underestimation and overestimation of the mixed noise parameters are undesirable. Underestimation is crucial for simple structure images and overestimation is undesired for complex structure images.
We have checked other test images and other noise parameter sets. Analysis carried out for all images in TID2008 has shown that reduction of PSNR and PSNR-HVS-M by about 0.5 dB approximately corresponds to MSSIM reduction by 0.005. Note that all three dependences are nonlinear and there is no strict relationship between them. The only observation is that if PSNR-HVS-M decreases, MSSIM usually diminishes as well and vice versa.
Thus, our approach to analysis consists in the following. The first task is to determine 2D areas of δ_{ V } and δ_{ k } where reduction is smaller than 0.5 dB according to the metrics PSNR and PSNR-HVS-M or smaller than 0.005 according to MSSIM for each considered image and each set of mixed noise parameters. It is assumed that if the mixed noise parameters’ estimates are within these areas, the estimation errors do not essentially influence the filtering accuracy.
Note that in addition to three aforementioned sets of mixed noise parameters for the model (1), we consider below one set (combination) of ${\sigma}_{\mathrm{si}}^{2}$ and k for the model (2) to simulate the real-life situation for which the multiplicative noise is dominant. Then, at the second stage, the obtained areas have to be aggregated by AND rule to provide final requirements to estimation accuracy under assumption that neither noise statistics nor image properties are available in advance.
4. Analysis of results
Simulation data for the test image #3, noise model (1), PSNR metric
k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR (δ_{ V } = -1, δ_{ k } = -1) | PSNR (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR} | δ _{V max} | δ _{k max} |
---|---|---|---|---|---|---|---|
0.2 | 10 | 32.66 | 32.99 | 38.64 | 38.65 | 0.3 | -0.1 |
0.2 | 50 | 72.62 | 29.52 | 36.58 | 36.62 | 0.4 | -0.7 |
1.0 | 30 | 144.01 | 26.55 | 34.52 | 34.57 | 1 | -0.2 |
As it is seen, for two cases (k = 0.2, ${\sigma}_{\mathrm{si}}^{2}$ = 10 and k = 1, ${\sigma}_{\mathrm{si}}^{2}$ = 30), the SD noise component is dominant (MSE_{inp} >2 ${\sigma}_{\mathrm{si}}^{2}$). For the case k = 0.2, ${\sigma}_{\mathrm{si}}^{2}$ = 50, the SI noise component is dominant. The first observation is that if noise is more intensive (compare the case k = 1, ${\sigma}_{\mathrm{si}}^{2}$ = 30 to the case k = 0.2, ${\sigma}_{\mathrm{si}}^{2}$ = 10), the DCT-based filtering is more efficient (PSNR(δ_{ V } = 0, δ_{ k } = 0) differs more from the corresponding PSNR(δ_{ V } = -1, δ_{ k } = -1)). In particular, PSNR(δ_{ V } = 0, δ_{ k } = 0) is larger by about 8 dB than PSNR(δ_{ V } = -1, δ_{ k } = -1) for the case k = 1, ${\sigma}_{\mathrm{si}}^{2}$ = 30.
If the SD noise component is dominant (see the plots in Figure 5a, c), it does not matter too much how accurate the estimates ${\widehat{\sigma}}_{\mathit{si}}^{2}$ are. It is considerably more important how accurate is the estimate of the SD noise parameter k. If ${\widehat{\sigma}}_{\mathit{si}}^{2}$ is quite accurate (let us say, |δ_{ V }| ≤ 0.5), then |δ_{ k }| should be smaller than about 0.4. Thus, if the SD noise component is dominant, then the requirement to accuracy of k is stricter than the requirement to accuracy of ${\widehat{\sigma}}_{\mathit{si}}^{2}$. This appears itself in the fact that a red surface ‘strip’ that is ‘over’ the corresponding threshold (black color surface) is oriented more parallel to the axis δ_{ V }.
Another situation is observed if SI noise component is dominant (see the plot in Figure 5b). Then accuracy of estimating the parameter k is less important, but the requirement to estimation of ${\sigma}_{\mathrm{si}}^{2}$ is stricter. In fact, for the considered case, it is desirable to provide |δ_{ V }| less than 0.3…0.4. The red surface ‘strip’ is oriented more parallel to the axis δ_{ k }. Thus, initial conclusion is quite trivial - it is necessary to more accurately estimate the parameter for the noise component which is dominant.
If one parameter is overestimated, then it is desirable to have another parameter underestimated ($\stackrel{\u2322}{k}<k$) to provide rather efficient filtering (see the values δ_{V max} and δ_{k max} in Table 1). It is better, if ${\stackrel{\u2322}{\sigma}}_{\mathit{si}}^{2}>{\sigma}_{\mathit{si}}^{2},{\delta}_{V}>0$. The worst case is if both parameters, ${\sigma}_{\mathrm{si}}^{2}$ and k, are underestimated. Then, undersmoothing is observed (look at data for δ_{ V } → -1, δ_{ k } → -1) and filtering efficiency is far from optimal (attainable). One more interesting observation that follows from analysis of data in Table 1 is that max_{PSNR} is only slightly (by 0.01…0.05 dB) larger than PSNR(δ_{ V } = 0, δ_{ k } = 0). This shows that the practical recommendation (5) works well enough.
Simulation data for the test image #3, noise model ( 1), PSNR-HVS-M metric
k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR-HVS-M (δ_{ V } = -1, δ_{ k } = -1) | PSNR-HVS-M (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR - HVS - M} | δ _{V maxvis} | δ _{k maxvis} |
---|---|---|---|---|---|---|---|
0.2 | 10 | 32.66 | 36.32 | 40.29 | 40.30 | 0.2 | -0.1 |
0.2 | 50 | 72.62 | 32.45 | 37.45 | 37.47 | 0.2 | -0.5 |
1.0 | 30 | 144.01 | 29.22 | 34.67 | 34.68 | 0.7 | -0.2 |
From the very beginning, let us stress that the values PSNR-HVS-M(δ_{ V } = -1, δ_{ k } = -1) are larger than the corresponding PSNR(δ_{ V } = -1, δ_{ k } = -1). This is due to the masking effects [39]. As it is according to the metric PSNR, PSNR-HVS-M increases for the recommended setting of the DCT filter parameter β (compare PSNR-HVS-M(δ_{ V } = 0, δ_{ k } = 0) to PSNR-HVS-M(δ_{ V } = -1, δ_{ k } = -1)). Visual quality improvement due to filtering is essential - from about 4 dB for non-intensive noise (k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 10), it reaches 5.5 dB for the case k = 1, ${\sigma}_{\mathrm{si}}^{2}$ = 30. Note that PSNR-HVS-M(δ_{ V } = 0, δ_{ k } = 0) for k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 10 exceeds 40 dB, i.e., residual distortions in the filtered image are almost invisible [35].
The observed values max_{PSNR- HVS- M} practically do not differ from the corresponding PSNR-HVS-M(δ_{ V } = 0, δ_{ k } = 0). Interestingly, all δ_{V maxvis} are larger than 0 while all δ_{k maxvis} are smaller than 0. This means that it is less risky to overestimate SI noise variance and underestimate the parameter k than to fall into other possible situations. Meanwhile, all δ_{V maxvis} (Table 2) are smaller than the corresponding δ_{V max} (see Table 1). This shows that for providing higher visual quality, it is undesirable to have considerable overestimation of mixed noise parameters. In some sense, it is equivalent to the recommendation to have β slightly smaller than 2.6 in threshold setting (5) to guarantee good visual quality of filtered image [38].
Simulation data for the test image #13, noise model ( 1), PSNR metric
k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR (δ_{ V } = -1, δ_{ k } = -1) | PSNR (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR} | δ _{V max} | δ _{k max} |
---|---|---|---|---|---|---|---|
0.2 | 10 | 32.0 | 33.08 | 33.91 | 33.96 | -0.6 | 0 |
0.2 | 50 | 71.93 | 29.56 | 30.99 | 31.09 | -0.3 | 0 |
1.0 | 30 | 140.24 | 26.66 | 28.64 | 28.77 | -0.6 | -0.1 |
Simulation data for the test image #13, noise model ( 1), PSNR-HVS-M metric
k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR-HVS-M (δ_{ V } = -1, δ_{ k } = -1) | PSNR-HVS-M (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR - HVS - M} | δ _{V maxvis} | δ _{k maxvis} |
---|---|---|---|---|---|---|---|
0.2 | 10 | 32.0 | 40.08 | 40.36 | 40.45 | -0.1 | -0.4 |
0.2 | 50 | 71.93 | 34.90 | 35.42 | 35.56 | -0.4 | 0 |
1.0 | 30 | 140.24 | 31.30 | 32.01 | 32.24 | -0.7 | -0.2 |
At the first glance, the dependences in Figures 8 and 9 are quite similar to the corresponding dependences in Figures 5 and 6. However, there are several distinctive differences. One of them is that filtering of the test image #13 is considerably less efficient than of the test image #3. Only for the most intensive noise (k = 1; ${\sigma}_{\mathrm{si}}^{2}$ = 30), the difference between PSNR(δ_{ V } = 0, δ_{ k } = 0) and PSNR for the original image (PSNR(δ_{ V } = -1, δ_{ k } = -1)) reaches 2 dB. This difference is only 0.83 dB for the case k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 10. The value max_{PSNR} is 0.05…0.13 dB larger than the corresponding PSNR(δ_{ V } = 0, δ_{ k } = 0), and this takes place if δ_{V max} < 0, i.e., if SI noise variance is underestimated and the parameter k is estimated properly (without error).
Overestimation of both parameters is severely undesirable since this leads to reduction of filtering efficiency and image oversmoothing (analyze data for δ_{ V } → 1, δ_{ k } → 1 where the dependences PSNR(δ_{ V }, δ_{ k }) rapidly decrease if δ_{ V } and δ_{ k } increase).
Analysis of plots in Figure 9 and data in Table 4 leads to the same conclusions. Note that in the case k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 10, PSNR-HVS-M increase due to filtering is only 0.28 dB, i.e., it is practically invisible. Only for k = 1; ${\sigma}_{\mathrm{si}}^{2}$ = 30 the value PSNR-HVS-M(δ_{ V } = 0, δ_{ k } = 0) is by 0.71 dB larger than for the original image, i.e., there is a small noticeable improvement of visual quality. This means that if filtering is applied to improve visual quality of highly textural images based on mixed noise parameters estimated in a blind manner, then these estimates should not be essentially larger than true values of these parameters (i.e., in fact, it is desirable to have δ_{ V } ≤ 0, δ_{ k } ≤ 0).
To show that the results obtained for other color components are similar to the results obtained for the red component, Figure 11d presents intersection areas for the green component according to the metric PSNR. Comparison of the obtained acceptable areas to the corresponding areas presented in Figure 11a shows that there is no essential difference.
Simulation data for the test image #3 and 13, noise model ( 2), PSNR metric
Image no. | k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR (δ_{ V } = -1, δ_{ k } = -1) | PSNR (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR} | δ _{V max} | δ _{k max} |
---|---|---|---|---|---|---|---|---|
3 | 0.01 | 20 | 171.67 | 25.78 | 33.66 | 33.70 | 0.6 | 0 |
13 | 0.01 | 20 | 168.90 | 25.85 | 28.02 | 28.10 | -0.6 | -0.1 |
Simulation data for the test image #3 and 13, noise model ( 2), PSNR-HVS-M metric
Image No | k | ${\mathbf{\sigma}}_{\mathsf{si}}^{\mathbf{2}}$ | MSE_{inp} | PSNR-HVS-M (δ_{ V } = -1, δ_{ k } = -1) | PSNR-HVS-M (δ_{ V } = -0, δ_{ k } = -0) | max_{PSNR - HVS - M} | δ _{V maxvis} | δ _{k maxvis} |
---|---|---|---|---|---|---|---|---|
3 | 0.01 | 20 | 171.67 | 28.46 | 33.48 | 33.49 | 0.6 | -0.1 |
13 | 0.01 | 20 | 168.90 | 30.32 | 31.24 | 31.39 | -0.8 | -0.2 |
The data are given in Tables 5 and 6, respectively. In this case, PSNR and PSNR-HVS-M improvement for accurately estimated k and ${\sigma}_{\mathrm{si}}^{2}$ (see data for δ_{ V } = 0, δ_{ k } = 0) is considerably smaller than for the test image #3. The requirement to accuracy of the parameter k estimation is stricter than to estimation accuracy of the parameter ${\sigma}_{\mathrm{si}}^{2}$ although the estimation errors with |δ_{ k }| ≤ 0.2 are still acceptable. Overestimation of the parameter k is especially undesirable.
A shortcoming of the analysis carried out above is that only two test images (although with essentially different properties) and only four (totally) sets of mixed noise parameters (although quite different) have been considered. Thus, let us also study two extreme cases. According to the previous analysis, the strictest restrictions are observed for the following cases: (a) a complex structure image corrupted by non-intensive noise with obvious dominance of one component, for example, additive; (b) a simple structure image corrupted by intensive noise with one prevailing component, for example, signal dependent of the model (2).
For the case a, we have used the test image #1 from the database TID2008 (Figure 1). The model (1) noise has been simulated with k = 0.01 and ${\sigma}_{\mathrm{si}}^{2}$ = 25. These settings relate to, e.g., the practical case of noisy (junk) component images acquired by old generation hyperspectral sensors [8]. For the case b, the test image #23 from TID2008 (Figure 1) that is one of the simplest has been exploited. The model (2) noise has been generated with k = 0.1 and ${\sigma}_{\mathrm{si}}^{2}$ = 10. Such noise can be observed in images formed by multi-look synthetic aperture radars.
Clearly, the way we followed in our analysis is not the only one possible. It is also possible to study tolerance of filtering techniques to ambiguity or inaccuracy of available a priori information in other ways. One of them can be based on simulating some estimates of noise parameters and using them instead of true values in denoising with statistical assessment of filtering efficiency. Following this way, we have assumed unbiased estimation of both parameters for the noise model (1) and estimates modeled as $\begin{array}{cc}\hfill \widehat{k}=k+\mathit{\Delta k},\hfill & \hfill {\widehat{\sigma}}_{\mathit{si}}^{2}={\widehat{\sigma}}_{\mathit{si}}^{2}+\Delta {\widehat{\sigma}}_{\mathit{si}}^{2},\hfill \end{array}$ where Δk and $\Delta {\widehat{\sigma}}_{\mathit{si}}^{2}$ are mutually independent zero mean random variables supposed to be Gaussian with standard deviations δ_{ k } and δ_{ v }, respectively. Then, varying δ_{ k }, δ_{ v }, it is possible to simulate erroneous estimation for a set of realizations, to determine what are filtering criteria values (PSNR, PSNR-HVS-M, MSSIM) for each realization and to statistically process them with obtaining mean and standard deviation for each considered metric as well as confidence intervals.
Note that erroneous estimation of noise parameters leads to degradation of all metrics (reduction of their mean values compared to the corresponding optimal ones). Because of this, we have determined confidence interval width as MCW = |MO - MM| + 3MS, where MO determines optimal value of a visual quality metric for errorless noise parameters estimates; MM is mean value of a visual quality metric; MS is standard deviation of a visual quality metric.
Simulation data for the test image #3 and 13, noise model ( 1), all considered metrics
δ_{ k }, δ_{ ν } | PSNR | PSNR-HVS-M | MSSIM | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
MO | MM | MS | MCW | MO | MM | MS | MCW | MO | MM | MS | MCW | |
Image 03 | ||||||||||||
Case k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 50 | ||||||||||||
0.05 | 36.58 | 36.57 | 0.013 | 0.046 | 37.40 | 37.40 | 0.006 | 0.017 | 0.9841 | 0.9841 | 0.0001 | 0.0002 |
0.10 | 36.55 | 0.045 | 0.160 | 37.38 | 0.030 | 0.103 | 0.9840 | 0.0002 | 0.0007 | |||
0.15 | 36.52 | 0.103 | 0.364 | 37.37 | 0.059 | 0.207 | 0.9839 | 0.0005 | 0.0016 | |||
0.20 | 36.49 | 0.158 | 0.557 | 37.34 | 0.091 | 0.328 | 0.9838 | 0.0007 | 0.0025 | |||
0.25 | 36.41 | 0.244 | 0.900 | 37.29 | 0.142 | 0.534 | 0.9834 | 0.0011 | 0.0041 | |||
Case k = 1.0; ${\sigma}_{\mathrm{si}}^{2}$ = 30 | ||||||||||||
0.05 | 34.53 | 34.52 | 0.020 | 0.065 | 34.67 | 34.67 | 0.004 | 0.018 | 0.9756 | 0.9764 | 0.0002 | 0.0005 |
0.10 | 34.51 | 0.051 | 0.173 | 34.66 | 0.026 | 0.095 | 0.9764 | 0.0004 | 0.0012 | |||
0.15 | 34.44 | 0.132 | 0.480 | 34.63 | 0.071 | 0.260 | 0.9760 | 0.0009 | 0.0031 | |||
0.20 | 34.37 | 0.336 | 1.162 | 34.58 | 0.189 | 0.662 | 0.9756 | 0.0023 | 0.0077 | |||
0.25 | 34.35 | 0.347 | 1.218 | 34.56 | 0.198 | 0.710 | 0.9755 | 0.0023 | 0.0079 | |||
Image 13 | ||||||||||||
Case k = 0.2; ${\sigma}_{\mathrm{si}}^{2}$ = 50 | ||||||||||||
0.05 | 31.00 | 31.00 | 0.034 | 0.102 | 35.42 | 35.42 | 0.046 | 0.138 | 0.9828 | 0.9828 | 0.0003 | 0.0008 |
0.10 | 30.98 | 0.066 | 0.215 | 35.40 | 0.087 | 0.277 | 0.9827 | 0.0005 | 0.0016 | |||
0.15 | 30.96 | 0.100 | 0.343 | 35.37 | 0.126 | 0.426 | 0.9825 | 0.0008 | 0.0025 | |||
0.20 | 30.97 | 0.151 | 0.479 | 35.40 | 0.194 | 0.598 | 0.9827 | 0.0012 | 0.0036 | |||
0.25 | 30.93 | 0.164 | 0.555 | 35.36 | 0.217 | 0.709 | 0.9825 | 0.0013 | 0.0043 | |||
Case k = 1.0; ${\sigma}_{\mathrm{si}}^{2}$ = 30 | ||||||||||||
0.05 | 28.64 | 28.64 | 0.041 | 0.127 | 32.01 | 32.01 | 0.054 | 0.164 | 0.9713 | 0.9713 | 0.0005 | 0.0017 |
0.10 | 28.64 | 0.080 | 0.246 | 32.01 | 0.101 | 0.308 | 0.9712 | 0.0010 | 0.0031 | |||
0.15 | 28.61 | 0.109 | 0.356 | 31.98 | 0.137 | 0.443 | 0.9710 | 0.0014 | 0.0045 | |||
0.20 | 28.62 | 0.143 | 0.450 | 32.00 | 0.185 | 0.563 | 0.9713 | 0.0019 | 0.0059 | |||
0.25 | 28.55 | 0.241 | 0.810 | 31.93 | 0.299 | 0.975 | 0.9706 | 0.0031 | 0.0098 |
5. Conclusions and future work
The question of influence of mixed noise parameters estimation accuracy on filtering efficiency in image enhancement applications is studied. It is demonstrated that the parameter that corresponds to a dominant noise type has to be estimated with a higher accuracy. This accuracy is characterized by a relative error that should be less than 20% for the dominant noise type and less than 30% for another type of noise. Then, decrease of filtering efficiency characterized by PSNR or PSNR-HVS-M drop compared to the optimum does not exceed 0.5 dB (MSSIM drop does not exceed 0.005), and thus, it is practically not noticeable (crucial). Note that these requirements practically coincide with requirements to accuracy of pure additive or pure multiplicative noise variance estimation [17] - an estimate has to differ from a true value by less than 20%. It is also important to stress that even the most advanced modern methods of blind estimation of mixed noise parameters do not always provide the required accuracy of parameters’ estimation [44].
It is also shown that for highly textural images it is better to have underestimation of the mixed noise parameters than overestimation. Unfortunately, it happens in practice that mixed noise parameters are usually overestimated for complex structure images [22]. This motivates a design of more accurate methods for blind estimation including analysis of dependences between the estimates of components of the mixed noise.
Besides, in the future, we plan to concentrate on considering spatially correlated noise which is rather typical in practice. Other filters based on using estimated parameters of mixed noise can be studied as well since restrictions for them can differ from restrictions obtained for the DCT-based denoising.
Declarations
Authors’ Affiliations
References
- Pratt WK: Digital Image Processing. 4th edition. New York: Wiley-Interscience; 2007. p. 807View ArticleGoogle Scholar
- Elad M: Sparse and Redundant Representations. vol. 20. In From Theory to Applications in Signal and Image Processing. Edited by: Elad M. Philadelphia, PA: Springer Science, Business Media, LLC; 2010:376.Google Scholar
- Al-Shaykh OK, Mersereau RM: Lossy compression of noisy images. IEEE Trans. on Image Process. 1998, 7(12):1641-1652. doi:10.1109/83.730376 10.1109/83.730376MathSciNetView ArticleGoogle Scholar
- Bovik A: Handbook on Image and Video Processing. Academic Press: Waltham, MA; 2000:1384.Google Scholar
- Donohо DL: De-noising by soft thresholding. IEEE Trans. on Information Theory 1995, IT-41(3):613-627.View ArticleGoogle Scholar
- Mallat S: A Wavelet Tour of Signal Processing. San Diego, CA: Academic; 1998:620.Google Scholar
- Sendur L, Selesnick IW: Bivariate shrinkage with local variance estimation. IEEE Sig. Process. Letters 2002, 9(12):438-441.View ArticleGoogle Scholar
- Lukin V, Abramov S, Ponomarenko N, Uss M, Zriakhov M, Vozel B, Chehdi K, Astola J: Methods and automatic procedures for processing images based on blind evaluation of noise type and characteristics. SPIE J. Adv. Remote Sens 2011. doi: 10.1117/1.3539768Google Scholar
- Foi A: Pointwise Shape-Adaptive DCT Image Filtering and Signal-Dependent Noise Estimation, Thesis for the degree of Doctor of Technology. Tampere: Tampere University of Technology; 2007.Google Scholar
- Oliver C, Quegan S: Understanding Synthetic Aperture Radar Images. SciTech Publishing: Raleigh, NC; 2004:479.Google Scholar
- Ponomarenko N, Lukin V, Egiazarian K, Lepisto L: Color image lossy compression based on blind evaluation and prediction of noise characteristics. In Proceedings of the SPIE 7870 of Image Processing: Algorithms and Systems VII. San Francisco; 23 Jan 2011.Google Scholar
- Liu C, Szeliski R, Kang SB, Zitnick CL, Freeman WT: Automatic estimation and removal of noise from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30(2):299-314.View ArticleGoogle Scholar
- Lim SH: Characterization of Noise in Digital Photographs for Image Processing. In Proceedings of the SPIE 6069 of Digital Photography II. San Jose; 15 Jan 2006.Google Scholar
- Curran PJ, Dungan JL: Estimation of signal-to-noise: a new procedure applied to AVIRIS data. IEEE Trans. Geosci. Remote Sens. 1989, 27: 620-628. 10.1109/TGRS.1989.35945View ArticleGoogle Scholar
- Uss M, Vozel B, Lukin V, Chehdi K: Local signal-dependent noise variance estimation from hyperspectral textural images. IEEE J. Select. Topics Signal Process 2011, 5: 469-486.View ArticleGoogle Scholar
- Aiazzi B, Alparone L, Barducci A, Baronti S, Marcoinni P, Pippi I, Selva M: Noise modelling and estimation of hyperspectral data from airborne imaging spectrometers. Ann. Geophys. 2006, 49: 19.Google Scholar
- Vozel B, Abramov S, Chehdi K, Lukin V, Ponomarenko N, Uss M, Astola J: Multivariate Image Processing. In Blind Methods for Noise Evaluation in Multi-Component Images. Edited by: Collet C, Chanussot J, Chehdi K. Wiley&Iste: Toulouse; 2010:263-302.Google Scholar
- Abramov S, Lukin V, Ponomarenko N, Egiazarian K, Pogrebnyak O: Influence of multiplicative noise variance evaluation accuracy on MM-band SLAR image filtering efficiency. vol 1. In Proceedings of the MSMW 2004. Kharkov; 21–26 Jun 2004:250-252.Google Scholar
- Kerekes JP, Baum JE: Hyperspectral imaging system modeling. Lincoln Laboratory J. 2003, 14(1):117-130.Google Scholar
- Barducci A, Guzzi D, Marcoionni P, Pippi I: CHRIS-Proba performance evaluation: signal-to-noise ratio, instrument efficiency and data quality from acquisitions over San Rossore (Italy) test site. Proceedings of the 3rd ESA CHRIS/Proba Workshop 21–23 Mar 2005.Google Scholar
- Abramov S, Zabrodina V, Lukin V, Vozel B, Chehdi K, Astola J: Improved method for blind estimation of the variance of mixed noise using weighted LMS line fitting algorithm. In Proceedings of the ISCAS. Paris; 30 May–2 Jun 2010:2642-2645.Google Scholar
- Abramov S, Zabrodina V, Lukin V, Vozel B, Chehdi K, Astola J: Methods for Blind Estimation of the Variance of Mixed Noise and Their Performance Analysis. In Numerical Analysis – Theory and Applications. Edited by: Awrejcewicz J. InTech: Richardson, TX; 2011:49-70. doi: 10.5772/1829Google Scholar
- Fevralev D, Lukin V, Ponomarenko N, Abramov S, Egiazarian K, Astola J: Efficiency analysis of color image filtering. EURASIP J. Adv. Signal Process. 2011, 2011: 41. doi:10.1186/1687-6180-2011-41 10.1186/1687-6180-2011-41View ArticleGoogle Scholar
- Oktem R, Egiazarian K: Transform domain algorithm for reducing the effect of film-grain noise in image compression. Electronic Letters 1999, 35(21):1830-1831. 10.1049/el:19991261View ArticleGoogle Scholar
- Ramponi G, D’Alvise R: Automatic Estimation of the Noise Variance in SAR Images for Use in Speckle Filtering. In Proceedings of the IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing. Antalya; 20–23 Jun 1999:835-838.Google Scholar
- Astola J, Kuosmanen P: Fundamentals of nonlinear digital filtering. Boca Raton: CRC Press LLC; 1997. 276Google Scholar
- Ponomaryov VI: Real time 2D-3D filtering using order statistics based algorithms. J. of Real-Time Image Process. 2007, 1: 173-194. 10.1007/s11554-007-0021-5View ArticleGoogle Scholar
- Lukin V, Ponomarenko N, Abramov S, Vozel B, Chehdi K: Improved noise parameter estimation and filtering of MM-band SLAR images. vol 1. In Proceedings of the MSMW 2007. Kharkov; 25–30 Jun 2007:439-441.Google Scholar
- Plataniotis KN, Venetsanopoulos AN: Color Image Processing and Applications. New York: Springer; 2000. 355View ArticleGoogle Scholar
- Astola J, Haavisto P, Neuvo Y: Vector median filter. Proc. IEEE 1990, 4: 678-689.View ArticleGoogle Scholar
- Touzi R: A review of speckle filtering in the context of estimation theory. IEEE Trans. on Geosci. and Remote Sens. 2002, 40(11):2392-2404. 10.1109/TGRS.2002.803727View ArticleGoogle Scholar
- Foi A: Clipped noisy images: heteroskedastic modeling and practical denoising. Signal Process. 2009, 89(12):2609-2629. 10.1016/j.sigpro.2009.04.035View ArticleGoogle Scholar
- Murtagh F, Starck JL, Bijaoui A: Image restoration with noise suppression using a multiresolution support. Astron. Astrophys. Suppl. Ser. 1995, 112: 179-189.Google Scholar
- Lukin V, Ponomarenko N, Abramov S, Vozel B, Chehdi K, Astola J: Filtering of radar images based on blind evaluation of noise characteristics. vol 7109. In Proceedings of the Image and Signal Processing for Remote Sensing XIV. Cardiff; 15–18 Sep 2008.Google Scholar
- Oktem R, Egiazarian K, Lukin VV, Ponomarenko NN, Tsymbal OV: Locally adaptive DCT filtering for signal-dependent noise removal. EURASIP J. Adv. Signal Process 2007. doi:10.1155/2007/42472Google Scholar
- Fevralev D, Krivenko S, Lukin V, Zelensky A, Egiazarian K: Speeding-up DCT based Filtering of Images. In CD ROM Proceedings of the Modern Problems of Radioengineering, Telecommunications and Computer Science (TCSET). Lviv-Slavsko; 23–27 Feb 2010.Google Scholar
- Dabov K, Foi A, Katkovnik V, Egiazarian K: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 2007, 16(8):2080-2095.MathSciNetView ArticleGoogle Scholar
- Lukin V, Ponomarenko N, Egiazarian K: HVS-Metric-Based Performance Analysis Of Image Denoising Algorithms. In Proceedings of the EUVIP. Paris; 4–6 Jul 2011.Google Scholar
- Ponomarenko N, Silvestri F, Egiazarian K, Carli M, Astola J, Lukin V: On between-coefficient contrast masking of DCT basis functions. In CD-ROM Proceedings of the VPQM. Scottsdale; 25–26 Jan 2007.Google Scholar
- Ponomarenko N: PSNR-HVS-M download page. 2009. . Accessed 28 Mar 2013 http://www.ponomarenko.info/psnrhvsm.htmGoogle Scholar
- Ponomarenko N, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F: TID2008 - a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics 2009, 10: 30-45.Google Scholar
- Lukin V, Zriakhov M, Krivenko S, Ponomarenko N, Miao Z: Lossy compression of images without visible distortions and its applications. In Proceedings of the ICSP 2010. Beijing; 24–28 Oct 2010:694-697.Google Scholar
- Wang Z, Simoncelli EP, Bovik AC: Multi-scale Structural Similarity for Visual Quality Assessment. vol 2. In Proceedings of the 37th IEEE Asilomar Conference on Signals, Systems and Computers. Monterey; 9–12 Nov 2003:1398-1402.Google Scholar
- Uss ML, Vozel B, Lukin V, Chehdi K: Image Informative Maps for Component-wise Estimating Parameters of Signal-Dependent Noise. J. Electron. Imag 2013., 22(1): doi:10.1117/1.JEI.22.1.013019Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.