Image enhancement using local intensity distribution equalization

Abstract

This paper proposes a local intensity distribution equalization (LIDE) method for image enhancement. LIDE applies the idea of histogram equalization to parametric model in order to enhance an image using local information. It reduces the amount of computational resources required by traditional method like the adaptive histogram equalization, but allows enhancing detail similar to the latter technique. Integral image was used to efficiently estimate local statistics needed by the parametric model. This data structure drastically reduces the computational cost especially for megapixel image where a large local window is preferred. It should be noted that, with a large local window, the intensity distribution could contain multiple peaks. LIDE can nicely handle such complex distribution via mixture of parametric models. To speed-up the mixture parameter estimation, we propose an EM algorithm that is also based on the integral image data structure. Experimental results show that LIDE produces an enhanced image with greater detail and lower noise compared to several existing methods.

1 Introduction

Histogram equalization (HE) is a widely used image enhancement technique. It increases the contrast of an image by transferring each grayscale value to a new one via the cumulative distribution function derived from the intensity histogram. For certain images containing varying lighting conditions, or those whose local detail is important, the global contrast enhancement by HE could be insufficient. Local contrast enhancement can be done by applying HE on the histogram computed in local window around each pixel. This is called local or adaptive histogram equalization (AHE) [1].

AHE is a computational intensive procedure even on today’s computer. Block-based HE is generally used to cut down the computational cost. Pizer et al. [2] apply block-based HE first to some locations in the image. Then an interpolation is used to determine the grayscale of other pixels. This reduces the computational cost at the expense of increasing blocking artifacts. Partially overlapped sub-block HE (POSHE) proposed by Kim et al. [3] reduces the blocking artifacts. Unfortunately, it does blur the detail of the image along the block boundaries as well. Lamberti et al. [4] suggest that POSHE procedure is, in fact, equivalent to applying the low-pass filter mask on the sub-region histogram. From this observation, they propose to perform POSHE calculation via the binomial filter. The computational cost of this method increases as function of the filter size. Cascade multi-step averaging procedure [5] has also been adopted to speed up the POSHE procedure. The resulting process is then faster than the original POSHE, but with the same enhancement capacity. The filtering idea is also used by Gauch [6] and Stark [7]. Both authors use several convolved images to compute local histogram. The computational cost resides mainly in the convolution phase which can be sped up in frequency domain. To speed up the calculation directly in spatial domain, Gonzalez and Woods [8, p. 103] suggest to carefully update local histogram since only one column and/or one row of the local window changes as it slides through the image. This idea is implemented later by Wang and Tao [9]. Other speed up techniques include multi-scale HE [10], parallel implementation [11], for example.

HE also suffers from its inability to preserve the brightness of the input image. For consumer electronics product such as video surveillance, preserving brightness is essential to avoid generating non-existing artifacts in the output image [12]. To overcome this problem, several research teams propose to partition the global histogram into two parts and apply HE to each sub-histogram independently. Kim [13] and Wan et al. [14] use mean and median of intensities as partition threshold respectively. Chen and Ramli [15] propose to select the threshold minimizing the absolute mean brightness error between the input and the resulting images. Singh and Kapoor [16] selects the threshold according to the exposure of the image. Recent work by Huynh-The et al. [17] carefully analyzes within-class variance of each interval in order to minimize the total squared error of each sub-histogram when equalizing them independently.

The histogram separation idea is further generalized by Chen and Ramli [18] and Sim et al. [19]. These two works rely on a recursive partition of the global histogram, but with different separation criteria. The two techniques produces a partition of the grayscale range into 2^r intervals where r is set by the user. More generic histogram separation is proposed by Wongsritong et al. [20], and later by Abdullah-Al-Wadud et al. [21] and Ibrahimet al. [22], These authors rely on either peaks or minima of the smoothed global histogram as partition criteria. In practice, we have found that the number of obtained sub-histograms depends on the smoothing procedure applied on the global histogram. Proper size of smoothing filter is needed in order to correctly remove local fluctuations for better enhancement. Menotti et al. [12] propose a more systematic histogram partition based on local discrepancy measures and a dynamic programming algorithm. This requires, however, a larger computational cost.

Other brightness preserving histogram-based enhancement techniques are based on histogram matching (HM) procedure. Wang and Ye [23] first search for the ideal target distribution maximizing the entropy while maintaining the average intensity of the input image. Then, HM is used to map the input histogram to this target distribution. A similar approach was proposed by Wang et al. [24]. However, the latter chooses the target distribution to be the one minimizing the difference to the uniform distribution subject to the average intensity preservation. In these two techniques, the optimization problem is solved analytically. The overall procedure is then very fast.

In this paper, we propose a variation of HE that can enhance local detail similar to AHE but with lower resources requirement. The basic idea is based on the use of parametric distribution instead of histogram in the equalization procedure. The proposed technique called local intensity distribution equalization (LIDE) relies on parametric cumulative distribution function computed around each pixel to enhance local detail. A data structure called integral image [25] is used to speed up the calculation especially for local statistics. This idea was also proposed recently by Liu et al. [26]. Liu et al. used Gaussian as local parametric model. The reported results showed advantages of this parametric equalization compared to existing methods. The authors observed that with a small local window, blocking artifact can be observed in the output image. A large local window could alleviate this problem, but the output will be very similar to that obtained from normal histogram equalization. The test images used in the work of Liu et al. were rather small (512×512 pixels) and the largest local window is equal to the size of the whole image. Experimentally, we have found that when working with a megapixel image, even with a large local window (500×500 pixels), one can still observe blocking artifact. In addition, with a large local window, the intensity distribution becomes more complex. It could contain multiple peaks, in which case the simple model is not adequate.

To handle complex distribution, a mixture model is more appropriate. The proposed LIDE can be used with local mixture model. The combination of LIDE with mixture model increases the memory requirement as well as the computational cost. However, compared to LIDE with simple model, it does produce output image with better quality including less blocking artifact. We also propose an EM algorithm that is adapted to take advantage of the integral image data structure for fast calculation.

It is worthy to be noted that the integral image can be used to speed-up the calculation of histogram in local window around each pixel that is needed for AHE as well. Hence, in the following, we shall first present this data structure in Section 2. Then, Section 3 introduces the LIDE method with simple probabilistic model (Section 3.1) and LIDE with mixture model (Section 3.2). Section 4 briefly describes how to enhance color images then Section 5 presents the experimental results. In Section 6, we discuss the brightness preserving issue using LIDE. Finally, Section 7 concludes this work.

2 Integral image

Given an image I, let I(x,y) be the value of pixel at position (x,y). The integral image of I, denoted by $\mathbb {I}$, is a two dimensional array in which each position (x,y) contains the sum of all pixel intensities above and to the left of (x,y) inclusive:

$$ \mathbb{I}(x,y)= \sum_{x'=0}^{x} \sum_{y'=0}^{y} I(x',y'). $$

((1))

With this array, the sum of pixel intensities in a local window of size (2d+1)×(2d+1) around a pixel (x,y) can be computed as follows:

$$\begin{array}{@{}rcl@{}} f(\mathbb{I},x,y,d) & = & \mathbb{I}(x_{0},y_{0})\,+\,\mathbb{I}(x_{1},y_{1})\! -\!\mathbb{I}(x_{0},y_{1})\,-\,\mathbb{I}(x_{1},y_{0}) \end{array} $$

((2))

$$\begin{array}{@{}rcl@{}} x_{0} & = & \max \{ 0, x-d-1 \} \end{array} $$

((3))

$$\begin{array}{@{}rcl@{}} y_{0} & = & \max \{ 0, y-d-1 \} \end{array} $$

((4))

$$\begin{array}{@{}rcl@{}} x_{1} & = & \min \{ W-1, x+d \} \end{array} $$

((5))

$$\begin{array}{@{}rcl@{}} y_{1} & = & \min \{ H-1, y+d \}, \end{array} $$

((6))

where W and H are the width and the height of the input image I ¹.

It is straightforward to adapt this procedure to compute mean and variance of the local window around each pixel. It should be emphasized that the direct calculation of local sum in the (2d+1)×(2d+1) window around each pixels requires a complexity of $\mathcal {O}(4WHd^{2})$ for W×H pixels image. With integral image, the complexity reduces to $\mathcal {O}(4WH)$. This complexity is constant and is independent from the window size. This means that the complexity of calculating the local average of 3×3 window and that of 300×300 window are the same.

The construction of an integral image can be done in a single pass over the input image using an additional accumulative array S. The following pair of recurrences is used in this calculation:

$$\begin{array}{@{}rcl@{}} S(x,y) & = & S(x,y-1)+I(x,y) \end{array} $$

((7))

$$\begin{array}{@{}rcl@{}} \mathbb{I}(x,y) & = & \mathbb{I}(x-1,y)+S(x,y) \end{array} $$

((8))

with S(x,−1)=0 and $\mathbb {I}(-1,y)=0$. The above construction algorithm can be seen as a special case of the wavefront propagation step used in construction of the integral histogram [27].

2.1 Integral image based AHE

We first observe that the histogram of intensities can be expressed as function of several summations, one for each grayscale level. Hence, we can also use the integral image data structure to reduce the computational cost of computing local histograms needed by AHE. Indeed, let I ^l be the image of grayscale level l defined as follows

((9))

where if a is true and 0 otherwise. The histogram h=h ₀,…,h ₂₅₅ in the local window of size (2d+1)×(2d+1) around (x,y) can be computed by:

$$ h_{l} = \frac{1}{Z} f(\mathbb{I}^{l}, x, y, d) $$

((10))

where $\mathbb {I}^{l}$ is the integral image of I ^l, and Z the normalizing constant such that $\sum _{l} h_{l} = 1$.

This approach was also used by Porikli [27] to fasten local histograms calculation. However, in that work, the author focused on extracting local histogram with multiple window sizes as signature for object matching.

This integral image based implementation is much faster than direct implementation of AHE, especially for large local window. However, it requires quite large memory since one integral image is needed for each grayscale level. An additional array is needed in the construction process of these integral images. Hence, the memory requirement is an order of $\mathcal {O}(257WH)$. For example, to process an image with 2000×2000 pixels, about 1 GB of memory is needed. This corresponds to about 4 M pixels image which is rather small compared to today’s mobile phone camera. The memory required by this procedure renders it impractical for megapixels images. In return, we can compute local histogram in local window of arbitrary size with the same computational cost. The next section describes the proposed LIDE method that can achieve similar performance as AHE but with much lower memory requirement.

3 Local intensity distribution equalization

The basic idea of LIDE is to use a parametric model instead of the traditional intensity histogram to cut down the memory requirement of AHE. Indeed, a parametric model, e.g., a Gaussian model, is constructed in the local window centered around each pixel (x,y). The cumulative distribution function (CDF) of this model is then used to compute the new grayscale value for the pixel (x,y).

3.1 LIDE with simple model

The CDFs of the two commonly used models namely Gaussian and Laplacian are as follows:

$$ \text{CDF}_{Gauss}(z,\mu,\sigma) = \frac{1}{2}\left[1+\text{erf}\left(\frac{z-\mu}{\sqrt{2\sigma^{2}}}\right)\right] $$

((11))

$${} \begin{aligned} \text{CDF}_{Laplace}(z,\mu,\sigma) &= \frac{1}{2}\left[{\vphantom{\left(1-\exp\left(-\frac{\sqrt{2}|z-\mu|}{\sigma}\right)\right)}}1 + \text{sign}(z-\mu)\right.\\ &\quad\times\left.\left(1-\exp\left(-\frac{\sqrt{2}|z-\mu|}{\sigma}\right)\right)\right] \end{aligned} $$

((12))

where $\text {erf}(z)=\frac {2}{\pi }{\int _{0}^{z}}\exp \left (-t^{2}\right)dt$ is the Gauss error function and μ and σ are mean and standard deviation (SD) of the sample, respectively.

Given an input image I, LIDE outputs image O is obtained using the following mapping:

$$ O(x,y)=255\times\text{CDF}\left(I(x,y),\mu(x,y,d),\sigma(x,y,d)\right) $$

((13))

where CDF(z,μ,σ) is CDF of either Gaussian (Eq. 11) or Laplacian (Eq. 12) model, μ(x,y,d) and σ(x,y,d) are local mean and SD in the local window around (x,y). Fortunately, the integral image technique can be used to efficiently compute these values. Let $\mathbb {I}$ and $\mathbb {I}^{2}$ be the integral image of the input image I and the squared intensity image, i.e., an image whose value at pixel (x,y) is I ²(x,y), we have:

$${} \mu(x,y,d) = \frac{1}{D}f(\mathbb{I},x,y,d) $$

((14))

$${} \sigma(x,y,d) = \min\left\{ \left(\frac{1}{D}f\left(\mathbb{I}^{2},x,y,d\right)-\mu^{2}\left(x,y,d\right)\right)^{\frac{1}{2}},\sigma_{\min}\right\} $$

((15))

$${} D = (x_{1}-x_{0}) \times (y_{1}-y_{0}) $$

((16))

where function f as well as x ₀, y ₀, x ₁, and y ₁ are given in the Eqs. 2 – 6, and σ _min is the minimum allowed SD in order to avoid instability in area of uniform intensity².

3.2 LIDE with mixture model

It is true that globally, the set of local simple models represent a complex model for intensity distribution modeling. However, with the increasing size of the local window, the simple model may not be sufficient to accurately represent the intensity distribution in each window. A mixture model is more appropriate tool to handle the multimodal distribution. This section describes how to use mixture model in the LIDE method as well as its training procedure.

Recall that the probability density function (PDF) of a mixture model with K components is of the form:

$$ p(z)=\sum_{k=1}^{K}w_{k}p(z|\mu_{k},\sigma_{k}) $$

((17))

where p(z|μ _k ,σ _k ) is the PDF of the component k, and w _k is its prior distribution. Two mixture models namely Gaussian mixture model (GMM) and Laplacian mixture model (LMM) are considered in this work. However, the proposed method can also be applied to other mixture model straightforwardly.

The CDF of this model can be computed as follows:

$$\begin{array}{@{}rcl@{}} \text{CDF}_{Mixture}(z) & = & \int_{-\infty}^{z}\sum_{k=1}^{K}w_{k}p(t|\mu_{k},\sigma_{k})dt \\ & = & \sum_{k=1}^{K}w_{k}\text{CDF}(z,\mu_{k},\sigma_{k}) \end{array} $$

where CDF(z,μ _k ,σ _k ) is the CDF of each component, e.g., CDF _Gauss or CDF _Laplace . The output image O obtained from the LIDE with mixture model is then given by:

$${} O(x,y)=255\times\left(\sum_{k=1}^{K}\!w_{k}\text{CDF}(I(\!x,y\!),\!\mu_{k}(x,y,d),\sigma_{k}(x,y,d))\!\!\right)\!. $$

((18))

3.2.1 EM algorithm using integral image

The above mapping function requires the parameters w _k ,μ _k ,σ _k k=1,…,K for all pixels in the image. Direct calculation of these parameters involves running an EM algorithm on the local window around each pixel in the image. Three nested loops are then needed; one loop over the pixels, one loop for the EM algorithm, and the innermost loop for computing local statistics in each iteration. Thus, the overall computational cost of direct implementation is huge.

To reduce the computational cost, we shall assume that the K means, μ ₁,…,μ _K , are shared across all pixels in the image, i.e., μ _k (x,y)=μ _k ,∀k,∀(x,y). This assumption implies that one needs to compute the difference between each pixel intensity and a mean value only once in order to update the SD of all components in every pixel.

Indeed, let $\mu _{k}^{(t)},k=1,\ldots,K$ be the shared K means at iteration t and let $w_{k}^{(t)}(x,y)$ and $\sigma _{k}^{(t)}(x,y)$ be the prior distribution and the standard deviation of component k of pixel (x,y), respectively. Initially, the K means are uniformly distributed between 0 and 255 ($\mu ^{(0)}_{k} = 255 k/K$). By default, we initialize $w_{k}^{(0)}(x,y)$ to 1/K and $\sigma _{k}^{(0)}(x,y)=255/K$.

The posterior probability of component k of pixel (x,y) or P _k (x,y) is computed as:

$$ P_{k}(x,y)=\frac{1}{Z_{k}}\times w_{k}^{(t)}p(I(x,y)|\mu_{k}^{(t)},\sigma_{k}^{(t)}(x,y)) $$

((19))

with Z _k a normalizing value to ensure that the posterior probabilities sum up to 1. Given P _k (x,y), the shared K means are updated as:

$$ \mu_{k}^{(t+1)}=\frac{\sum_{(x,y)}I(x,y)\, P_{k}(x,y)}{\sum_{(x,y)}P_{k}(x,y)}. $$

((20))

Once the K means, μ ₁,…,μ _K , are updated, the standard deviations are recomputed via the integral image $\mathbb {D}_{k}$ of the following squared difference image D _k :

$$ D_{k}(x,y)=P_{k}(x,y)\,\left(I(x,y)-\mu^{(t+1)}_{k}\right)^{2}. $$

((21))

The SD update equation is as follows:

$$ \sigma_{k}^{(t+1)}(x,y)=\min\left\{ \left(\frac{f(\mathbb{D}_{k},x,y,d)}{f(\mathbb{P}_{k},x,y,d)}\right)^{\frac{1}{2}}\:,\:\sigma_{\min}\right\} $$

((22))

where $\mathbb {P}_{k}$ is the integral image of the posterior probability image. The prior probability is also updated from $\mathbb {P}_{k}$, i.e.,

$$ w_{k}^{(t+1)}(x,y)=\frac{1}{D}f(\mathbb{P}_{k},x,y,d), $$

((23))

where D is given in the Eq. 16. The proposed EM algorithm using integral image is summarized in Algorithm 1.

3.3 Complexity

For simple models like Gaussian and Laplacian, two integral images are needed, one to compute the local mean and another one for local SD. In the current implementation, two passes are needed for LIDE with a simple model. The first pass computes the two integral images, and the second one computes the new grayscale value using the local CDF. The computational cost is then an order of $\mathcal {O}(2WH)$. However, this means that the two integral images are constructed in the same time. Hence, two additional array (S in the Eq. 8) are needed. The memory requirement is thus an order of $\mathcal {O}(4WH)$.

For LIDE, with mixture of K components, 3K images of size W×H are needed for storing σ _k ,w _k ,P _k k=1,…,K. For each component, the standard deviation update (Eq. 22) and the prior update (Eq. 23) require 2 and 1 additional integral images, respectively. As these updates can be done sequentially, the whole process requires additional $\mathcal {O}(4WH)$ memory at maximum. The K means μ ₁,…,μ _K require only K floating point values which are negligible compared to the size of the image. The memory requirement of LIDE with mixture model is then an order of $\mathcal {O}((3K+4)WH).$

The computational cost of this mixture model is dominated by the EM algorithm. For each iteration, 4K passes over the image are needed for the four steps a–d in the Algorithm 1. The overall complexity is thus an order of $\mathcal {O}(4KWHT)$, with T the number of iterations in the EM algorithm. In practice, ten maximum iterations are used by default.

4 Color image enhancement

For color image, we first apply the enhancement method to its grayscale version. Then use the ratio between the old and new grayscale value to adjust the 3 channels red, green, blue. Let C be color image with C(x,y).r, C(x,y).g, and C(x,y).b be the red, green, and blue component of the pixel (x,y). We first convert the input C image into grayscale image I as follows:

$$ I(x,y) = \frac{1}{3} \left(C(x,y).r + C(x,y).g + C(x,y).b\right) $$

((24))

Let O be the enhanced version of I obtained from an enhancement method (HE, AHE, LIDE, etc.), and $\tilde {C}$ be the enhanced version of C. The pixel value of $\tilde {C}$ is given by:

$$\begin{array}{@{}rcl@{}} \tilde{C}(x,y).r & = & \alpha \;C(x,y).r \\ \tilde{C}(x,y).g & = & \alpha \;C(x,y).g \\ \tilde{C}(x,y).b & = & \alpha \;C(x,y).b \end{array} $$

with α=O(x,y)/I(x,y).

We have also tested other methods such as enhancing the three channels R, G, and B independently or enhancing only the channel V in HSV colorspace, etc. However, these methods do not always preserve the correct color compared to the original image. The method described above is simple, yet produces correct color. Thus, it will be used in the remainder of this paper.

5 Experiments

5.1 Experimental setup

The complexity analysis in Section 3.3 shows that LIDE requires less memory and computational cost compared to AHE. In this Section, we would like to measure the actual improvement of LIDE. In addition, we also consider the quality of the output image when parametric model is used instead of the traditional histogram. To this end, we use photography images from MIT-Adobe FiveK dataset [28]. The size of images in this dataset is quite large, i.e., about 10 megapixels. For these images, the size of the local window was empirically set to 501×501 (d=200). Indeed, a too small local window could also enhance noise whereas a too large window would fail to enhance local detail. The selected size yields good result in many tests for both AHE and LIDE.

Integral image based AHE was used in this experiment. It should be noted that Pizer et al. [2] suggest that in the histogram building process, one should clip large bins, and distribute the count into small bins of the histogram. This procedure, known as contrast limited AHE (CLAHE), allows reducing noise amplification, especially from uniform area. In our implementation, we used relative discounting strategy to clip the local histograms. Indeed, all histogram bins were discounted by the same proportion, and the sum of the reduced count was distributed equally amongst all bins. Five percent discount was used throughout this work.

Ten components were used by default for both the Gaussian mixture model (GMM) and Laplacian mixture model (LMM) in LIDE. Hence, LIDE with mixture model would require roughly one seventh of memory needed by integral image based AHE. A less number of components could reduce the memory requirement as well as the processing time, but it will also reduce the local detail enhancement. Figure 1 shows an example of zoom-in portion of the original image in Fig. 2 as well as portions from images processed by LIDE with LMM using 3, 5, 10, and 20 components. We have experimentally found that 10 components per mixture is a good trade-off between detail enhancement and the resources requirement in several cases.

Fig. 1

A zoom-in portion of image from Fig. 2 a as well as from the image processed by LIDE with LMM using 3, 5, 10, and 20 components in b, c, d, and e, respectively

Fig. 2

An example of image from MIT-Adobe FiveK dataset (a) and the output from AHE (b) and LIDE (c–f) as well as from other enhancement techniques (g–m)

Four variations of LIDE were considered namely LIDE with simple Gaussian model (LIDE-G), LIDE with simple Laplacian model (LIDE-L), LIDE with GMM (LIDE-GMM), and LIDE with LMM (LIDE-LMM). These four LIDE variations were compared against other existing techniques namely the dynamic histogram equalization (DHE) [21], the multi-histogram equalization (MHE) [12], the exposure-based sub image histogram equalization (ESIHE) [16], the brightness preserving histogram equalization with maximum entropy (BPHEME) [23], the flattest histogram specification with accurate brightness preservation (FHSABP) [24], and the histogram equalization with GMM (HEGMM) [29]. These techniques rely on the global histogram. Thus, we shall refer to them as global enhancement methods.

For DHE, a smoothing filter of size 3 was used. We applied this filter recursively until the distance between successive local minima is larger than a predefined value (20 in this experiment). For MHE, the middle value of each interval was used in the discrepancy calculation, and the weight coefficient (the parameter ρ in [12]) equals to 0.8 was used. In this experiment, we evaluated the number of intervals partition from 5 to 10, and 6–7 intervals were found to be optimal for most images.

HEGMM also relies on a Gaussian mixture model but with different approach from ours. Indeed, HEGMM first constructs a GMM modeling the global histogram of the intensities. Then, it partitions the grayscale range of the input image by analyzing the intersection between Gaussian components. For each obtained interval, HEGMM identifies the dominant Gaussian component. Once partitioned, HEGMM constructs the mapping function such that the CDF of the distribution in the output interval is preserved.

Our implementation of HEGMM also relied on the component-wise EM algorithm for mixtures algorithm [30] as used by Celik and Tjahjadi [29]. By default, the mixture model was initialized with 20 components. Then, each component was updated sequentially. The component whose prior probability falls below 0.001 was discarded.

Nomenclature of methods evaluated in this work is summarized in Table 1. Different enhancement techniques were carefully implemented in C++ using OpenCV library³. All tests were performed on a MacBook Pro 2.4 GHz Intel Core i5.

Table 1 Nomenclature of methods evaluated in this work

5.2 Qualitative results

Example 1 is shown in Fig. 2 along with the output from different enhancement techniques. For this image, AHE and LIDE allow enhancing local detail in the water that cannot be seen in the original image Global enhancement methods can also bring up some detail in the water but not all. In fact, several global enhancement methods barely enhance the detail in this area, since they try to preserve the overall brightness of the input image.

For AHE, all details in the water, on the lotus leafs as well as the ripples on the water seem to be equally enhanced. For LIDE, one can still distinguish between underwater detail and the rest. LIDE with mixture model produces better detail enhancement as can be seen, for example on the lotus flower. The Laplacian model seems to produce a better detail enhancement compared to the Gaussian model for both simple and mixture cases. The output from AHE seems to be the most oversaturated, followed by the output from LIDE with simple model and with mixture model, respectively.

Moreover, one can observe a shadow-like effect around the border of lotus leafs in the output of AHE as well as that of LIDE-G and LIDE-L. Recall that AHE and LIDE enhance each local regions independently. Hence, if the connected regions had distinctive intensities then these techniques will produce transition area between the two regions. This transition area corresponds to the border effect that can be observed. This border effect is less visible in the output of LIDE with both mixture models.

For the memory requirement, AHE needs about 3 GB. LIDE method allows reducing the memory requirement significantly compared to AHE. Indeed, LIDE with mixture model requires about 415 MB. LIDE with simple model further cuts the memory requirement down to less than 50 MB. The runtime of AHE is almost 7 min. However, it should be emphasized that direct implementation without integral image would require much more than 7 min to perform the same enhancement. LIDE with mixture of 10 components requires about 3 min. LIDE with simple model needs less than 3 s, that is more than 130 times faster than AHE. The runtime of LIDE with simple model is almost the same as other global enhancement methods. The resources requirement for this example can be found in Table 2.

Table 2 Evaluation results for the first test image (Fig. 2). The size of the original image was 4386×2920 pixels

Example 2 is shown in Fig. 3. This image contains a strong contrast between light and shadow regions. The border effect can be observed again between these regions, especially on the orange cover book in the middle of the image. This effect is more emphasized in the output of LIDE with simple model. LIDE with mixture model produces less border effect, but still more visible compared to AHE. Global enhancement techniques do not yield such artifact, but most of them fail to enhance the detail in this image. In fact, only EISHE and HEGMM successfully enhanced details in the shadow area. They still fail for the bright area. Local techniques, i.e., AHE and LIDE, successfully boosted the detail in both dark and bright regions. They did, however, amplify noise in these regions as well. LIDE produces less noise compared to AHE, especially when using with the mixture model.

Fig. 3

An example of image from MIT-Adobe FiveK dataset (a) and the output from AHE (b) and LIDE (c–f) as well as from other enhancement techniques (g–m)

For this image, LIDE with mixture model requires memory less than one-seventh of that required by AHE. With simple model, LIDE reduces the memory requirement from almost 3 GB to below 50 MB. This is small enough to fit into today’s mobile device. Moreover, the runtime of LIDE with simple and mixture models are about 3 and 166 s respectively whereas the runtime of AHE is about 410 s. The proposed method is, again, faster than AHE. The resources requirement for this example can be found in Table 3.

Table 3 Evaluation results for the second test image (Fig. 3). The size of the original image was 4284×2844 pixels

Example 3 is shown in Fig. 4. This image is bright but with low contrast. AHE yields an oversaturated image as in previous examples. One can observe the border artifact in the powder inside the tubes in output images from LIDE with simple model as well as from AHE. LIDE with mixture model seems to be able to better handle this case.

Fig. 4

An example of image from MIT-Adobe FiveK dataset (a) and the output from AHE (b) and LIDE (c–f) as well as from other enhancement techniques (g–m)

We can also observe a staircase artifact in the left part of the output from LIDE-G and LIDE-L. Figure 5 a,b show this portion from Fig. 4 LIDE-G and LIDE-L. In fact, this staircase artifact happens when LIDE is used with simple model, especially in the region with a low variation of intensities. Increasing σ _min in the Eq. 15 could alleviate this problem. However, the local detail in other regions will also be smoothened.

Fig. 5

Staircase effect in the images output from LIDE-G (a) and LIDE-L (b), respectively

Interestingly, BPHEME and FHSABP that try to preserve brightness does overly brighten up the tubes in the foreground. Some local detail in the background seems to be enhanced by these methods as well. For this example, the output from LIDE seems to be flatter compared to these techniques. The resource requirement for this example can be found in Table 4.

Table 4 Evaluation results for the third test image (Fig. 4). The size of the original image was 5010×3336 pixels

Example 4 is shown in Fig. 6. For this image, all local enhancement methods amplify noise in the dark area of dog’s ear inside the car. In this case, the ear shape enhanced by AHE seems to be the best. The output from LIDE in this area seems to be pure random noises. Other global enhancement methods do not produce such artifacts but they do not increase the detail neither on the car’s window nor inside the car. The resources requirement for this example can be found in Table 5.

Fig. 6

An example of image from MIT-Adobe FiveK dataset (a) and the output from AHE (b) and LIDE (c–f) as well as from other enhancement techniques (g–m)

Table 5 Evaluation results for the fourth test image (Fig. 6). The size of the original image was 3068×2048 pixels

Example 5 is shown in Fig. 7. This image contains a bright area of the sky on top. AHE performs poorly on this area. Indeed, it amplifies local noises significantly as can be seen in Fig. 7 b. LIDE with simple model produces a staircase effect on this area as well. We have experimentally found that, generally, the staircase effect appears on a region with uniform intensity. Adjusting the size of the local window could potentially alleviate this problem. It is, however, beyond the scope of this work. Fortunately, working with local mixture model can also resolve this artifact problem. In fact, LIDE with mixture model does not amplify local noise either. Global methods failed to enhance detail in the wave and in the splashing water. The resources requirement for this example can be found in Table 6.

Fig. 7

An example of image from MIT-Adobe FiveK dataset (a) and the output from AHE (b) and LIDE (c–f) as well as from other enhancement techniques (g–m)

Table 6 Evaluation results for the fifth test image (Fig. 7). The size of the original image was 3040×2014 pixels

5.3 Quantitative results

5.3.1 Evaluation measures

Entropy. Apart from the visual inspection, we also computed some objective measure to assess the the quality of the enhanced image. Followed the work of Singh and Kapoor [16], the entropy was used to measure the quality of the equalization procedure:

$$ H(I) = - \sum_{l=0}^{255} P^{I} (l) \log P^{I} (l) $$

((25))

where P ^I is the histogram of intensity derived from the image I. Flatter distribution would correspond to higher entropy.

EBCM. High entropy does not imply better local enhancement of the image. Indeed, to measure the local detail enhancement, previous works rely on the edge-based contrast measure (EBCM) [29, 31]. EBCM is defined as function of the following edge-based contrast:

$$ c(x,y)=\frac{|I(x,y)-e(x,y)|}{|I(x,y)+e(x,y)|} $$

((26))

where

$$e(x,y)= \frac{\sum_{(i,j)\in\text{Neigh}(x,y)} I(i,j)m_{I}(i,j)}{\sum_{(l,k)\in\text{Neigh}(x,y)}m_{I}(l,k)} $$

with Neigh(x,y) the set of neighbor pixels of (x,y), and m _I (x,y) is the gradient magnitude at pixel (x,y). The EBCM of an image I is given by:

$$ EBCM(I)=\sum_{(x,y)}c(x,y) $$

((27))

Image with higher contrast is expected to have larger EBCM value.

GradMag. Experimentally, we have found that in several cases EBCM produces similar values that can be hardly distinguished. Thus, in this work, we consider another measure that is the average of gradient magnitude. Indeed, good local enhancement technique should increase the magnitude of edges in the output image. It is true that high gradient magnitude could correspond to noise. Besides, we have observed that local enhancement techniques tend to amplify noise. Hence, we also consider measuring the noise level in the output image as described below.

aPSNR. The classical noise level measuring is the peak signal-to-noise ratio (PSNR) that is computed from the mean squared error (MSE) between the original noise-free image and the processed image. Low noise image would correspond to high PSNR value; noisy image would have low PSNR.

It should be noted that PSNR cannot always distinguish noise injection from an enhancement, supposing that we slightly shifted the intensity of all pixels in a black image uniformly toward 255. In this case, we did not introduce any noise into the image. However, the resulting image will have high MSE, thus low PSNR value as if we inject some noise with large amplitude into the original black image. Hence, PSNR does not measure only the noise level in the processed image but also its difference from the original image. As local enhancement techniques do not aim at preserving the intensity of the original pixels, the output image could have high MSE when compared to the input image. This does not necessarily mean that it has high level of noise as one might expect.

To alleviate this problem, we consider the local mean image as reference instead of the original image in MSE calculation. The resulting MSE will be used to computed the approximate PSNR (aPSNR) as follows:

$$\begin{array}{@{}rcl@{}} aMSE & = & \frac{1}{WH} \sum_{(x,y)} \left(I(x,y) - \mu(x,y,d) \right)^{2} \\ aPSNR & = & 20 \log_{10} (255) - 10 \log_{10}(aMSE) \end{array} $$

((28))

where d is the size of local window as in AHE and LIDE calculation. Image with low noise should have low aPSNR as well.

5.3.2 Experimental results

Tables 2, 3, 4, 5, and 6 summarize different measures obtained from the output of enhancement methods for each example as well as their runtime and memory requirement. Global enhancement techniques requires only small amount of additional memory that is less than the size of the input image. Hence, their memory requirement can be considered negligible.

LIDE methods give high entropy compared to several global enhancement methods but less than AHE. In fact, for these images, the simple HE already yields high entropy. Most global enhancement methods yield lower entropy. This is due to the fact that these methods try to preserve the brightness of the original image. They are constrained to output image whose brightness is close to the input image. Hence, only few details can be enhanced.

To measure the detail enhancement, the obtained EBCM values are very close in several cases. The gradient magnitude is easier to be distinguished. AHE yields the highest gradient magnitude in all cases. It also yields very low aPSNR as one might expected since AHE tends to boost up noise. Recall that local enhancement methods process each local region regardless of the global histogram. Consequently, the dark area in the input image will be enhanced similar to other areas. Hence, there is no real dark area in output image from these techniques. The dark intensities are mostly distributed to edges and to noises in the output image. This is why AHE produces image with high noise level. LIDE aims at local enhancement also yields high noise level. LIDE-L and LIDE-G yield the second and the third lowest aPSNR in most cases. With mixture model, LIDE outputs higher aPSNR, i.e., lower noise level images. In fact, LIDE with mixture model yields comparable aPSNR as global enhancement methods, even higher in some cases. This is in agreement with earlier visual inspection that LIDE with mixture model produces less speckle noises compared to AHE.

5.4 Discussion

The above examples illustrate that the proposed LIDE with mixture model, especially with LMM, allows enhancing local detail similar to AHE, but with less noise amplification. It requires much less memory compared to integral image-based AHE. This makes LIDE-LMM practical for megapixel images. We can further reduce the resources requirement by using LIDE with simple model (e.g., LIDE-G or LIDE-L). It can be easily implemented on limited resource devices such as mobile phones. However, it may introduce artifacts such as the blocking and the staircase effects.

We believe that LIDE with mixture model amplifies less noise thanks to the EM-training procedure. Indeed, the EM algorithm spreads the information of each pixel amongst all components. This is similar to the histogram clipping and the redistribution of counts in contrast limited HE.

6 Brightness preservation

For some applications, brightness preservation is a required property in order to avoid abrupt change in the visual brightness and other undesirable artifacts. Preserving the brightness also prevents us from amplifying noises in both too dark and too bright areas. An interesting approach by Wang and his colleagues [23, 24] relies on histogram matching (HM) to transform the CDF of an input intensity distribution to the desired target distribution. The target distribution is obtained from a constraint optimization problem where the constraint is to preserve the average intensity, i.e., the average of the target distribution must match the average intensity of the input image. Wang et al. proposed two methods namely BPHEME [23] and FHSABP [24] that differ on the objective function of the optimization problem. Due to the discrete nature of the histogram, the obtained image could have slightly different average intensity compared to the input image.

The above approach can be applied with LIDE straightforwardly. We start by computing the target distribution (either BPHEME or FHSABP) using global intensity histogram. Then HM is used in each local window to map between local histogram and the target distribution. We have found that this modified LIDE produces image with brightness level closer to that of input image compared to normal LIDE. However, as the local equalization is performed independently around each pixel, the global intensity preserving property cannot be guaranteed. We are working on a better way to combine the intensity constraint with local enhancement method.

Figure 8 shows an image and its output from BPHEME, FHSABP, as well as LIDE with LMM and its constrained version using BPHEME and FHSABP target distributions. The average intensity of each image is depicted below it. The normal BPHEME and FHSABP produce average intensity very close to the input image. Constrained LIDE produces larger difference but yield better detail image. In this case, the normal LIDE with LMM overamplifies noise in the dark area at the bottom of the image. By constraining the output to preserve the brightness, the visibility of these noise can be reduced.

Fig. 8

An example of image from MIT-FiveK dataset (a) and the outputs from several enhancement techniques; b LIDE-LMM with 3 components, c LIDE-LMM with 10 components, d BPHEME e constrained LIDE-LMM-BPHEME with 3 components, f constrained LIDE-LMM-BPHEME with 10 components, g FHSABP h constrained LIDE-LMM-FHSABP with 3 components, i constrained LIDE-LMM-FHSABP with 10 components

One can observe that increasing the number of components in the mixture does reduce the intensity of the brightest spot in the image, but increase the visibility in the dark area instead. As a result, the distribution of intensities will differ from the target distribution more and more. In fact, with more components, the obtained intensity histogram from either BPHEME constraint or FHSABP constraint appear to be very close. Figure 9 shows the target distributions for BPHEME and FHSABP of Fig. 8 a as well as the intensity histogram obtained from constrained LIDE-LMM. Hence, we would suggest LIDE-LMM with small number of components (3 or 5) for brightness preservation using either BPHEME or FHSABP target distribution.

Fig. 9

a The BPHEME target distribution of Fig. 8 a and the intensity histogram from BPHEME (b) and BPHEME constrained LIDE-LMM with 3 components (c) and with 10 components (d). e The FHSABP target distribution of Fig. 8 a and the intensity histogram from FHSABP (f) and FHSABP constrained LIDE-LMM with 3 components (g) and with 10 components (h)

7 Conclusion and future works

This paper proposes a variation of the adaptive histogram equalization technique called LIDE that is based on parametric model instead of the traditional histogram of intensities. This allows drastically reduce the memory requirement. We also propose an efficient implementation of an EM algorithm for LIDE with mixture model. Experimental results show that it is more than 2 times faster than a carefully implemented AHE. We believe that the integral image based EM algorithm could be applied to other tasks involving mixture model such as the GMM background modeling [32] in video processing.

LIDE with mixture model, especially LMM, produces image with clear detail and less speckle noise compared to AHE. It is true that local enhancement methods may not be suitable for all cases. However, we believe that if local enhancement is needed, the proposed method represents a good and interesting alternative to AHE.

It is true that local enhancement could lead to noise amplification. Experimental results suggest that LIDE can handle this problem better than AHE. An interesting method to reduce this artifact is due to Saleem et al. [33]. The idea is to fuse the results from global and local enhancement method. This corresponds to fusing enhancement results from multiple scales. We believe that the mixture model used in LIDE can be extended to handle this multi-scale enhancement approach nicely. This is currently under investigation.

Finally, both AHE and LIDE can handle the change in lighting condition to some extent. Indeed, information contained in too dark or too bright areas are not sufficient for correctly enhancing these regions. Noise amplification can be easily observed on these regions. Brightness preservation constraint could be used to conceal such artifact. However, a better way to handle this case would be to fuse information from images obtained under different exposures. We believe that the local mixture model used in LIDE may provide an interesting method for fusioning multi-exposure images. This will be further investigated as well.

8 Endnotes

¹ This means that we consider pixels outside the input image as having zero intensity.

² We have experimentally found that setting D=(2d+1)² would produce artifacts on border pixels where local window fall out of the input image.

³ http://opencv.org