Segmentation-based image defogging using modified dark channel prior

Abstract

Image acquisition under bad weather conditions is prone to yield image with low contrast, faded color, and overall poor visibility. Different computer vision applications including surveillance, object classification, tracking, and recognition get effected due to degraded hazy images. Dehazing can significantly improve contrast, balance luminance, correct distortion, remove unwanted visual effects/ and therefore enhance the image quality. As a result, image defogging is imperative pre-processing step in computer vision applications. Previously, dark channel prior-based algorithms have proven promising results over the available techniques. In this paper, we have proposed a modified dark channel prior that uses fog density and guided image-filtering technique to estimate and refine transmission map, respectively. Guided image filter speeds up the refinement of transmission map, hence reduces the overall computational complexity of algorithm. We have also incorporated segmentation of the foggy image into sky and non-sky regions, after which, the modified dark channel prior and atmospheric light is computed for each segment. Then, the average value of atmospheric light for each segment is used to estimate transmission map. We have performed quantitative and subjective comparison for effective evaluation of our proposed algorithm against the current state-of-the-art algorithms on natural and synthetic images. Different quality metrics, such as saturation, mean square error, fog density, peak signal to noise ratio, structural similarity index metric, dehazing algorithm index (DHQI), full-reference image quality assessmen (FR-IQA), and naturalness of dehazed images have shown the proposed algorithm to be better than existing techniques.

1 Introduction

Outdoor images are degraded due to hazy weather conditions such as mist, rain, clouds, and fog. Image acquisition, during such weather conditions, reduces contrast of the image, fades colors, degrades quality, and makes object features difficult to perceive by humans and conventional computer vision systems. Therefore, it is imperative to perform image dehazing as it is required for different computer vision application such as surveillance, image classification, detection, tracking, recognition, and obtaining satellite imagery. An effective dehazing approach should be able to significantly improve contrast, balance luminance, correct distortion, remove unwanted visual effects, and enhance the overall image quality. Figure 1 shows two examples of the foggy and the corresponding fog-free images taken from the RESIDE dataset [1].

Fig. 1

Sample images taken from the RESIDE dataset [1]. a Foggy Images where the fog is generated synthetically. b Fog-free images

In the literature, image dehazing or defogging is achieved through image enhancement, image restoration, fusion-based approaches, and machine learning [2]. Image enhancement algorithms are used to modify color and contrast of the image without improving their quality, whereas image restoration-based techniques defog the image depending upon fog density [2].

Wang et al. have used wavelet transform and single scale retinex algorithm to correct the foggy images and improve brightness [3]. The depth map was computed by using Bayesian theory and Markov regularization to reduce halo artifacts in the images.

He et al. have used morphological operations to calculate fog density [4] in which, foggy image brightness is increased before estimating atmospheric light. In [5], adaptive histogram equalization is used for contrast enhancement. Similarly, Xu et al. proposed defogging algorithm based on contrast limited adaptive histogram equalization to improve contrast while removing fog and unwanted noise particles [6]. Various defogging algorithms based on polarization are proposed in [7, 8]. Schechner et al. have proposed defogging using inversion process of atmospheric scattering where two images are taken from a polarizer at different orientations [7]. In [8], Miyazaki et al. have proposed a polarization-based technique to remove the fog particles from the image. Their proposed algorithm takes two input images as a prior knowledge to compute parameters used for image defogging.

Tan et al. have proposed defogging algorithm considering dark channel of foggy images [9]. Dark channel contains the low contrast pixel values and it acts as a prior knowledge for foggy image restoration [10]. In [11], He at al. have proposed a novel dark channel prior (DCP) algorithm by assuming air light constraint for transmission map estimation. Another DCP-based algorithm is proposed in [12] that uses bilateral filtering technique to refine transmission map. Chen et al. have obtained clear dark channel and preserved edge information by using bilateral filter and the adaptive median filter [13]. The process of DCP-based algorithm consists of estimation of atmospheric light, estimation and refinement of transmission map, and consequently fog-free image reconstruction. In this method, foggy image brightness is increased prior to estimating atmospheric light. Binary tree-based atmospheric light estimation is proposed by Tang et al. in [14]. Image defogging using RGB, LAB, and HSV color spaces is proposed by Nair et al [15]. In this approach, transmission map is refined using surround filter and the surround constant is chosen arbitrarily. Li et al. have proposed an improved bilateral filtering technique for transmission map refinement [16]. In [17], Zhang and Hou have proposed K-means clustering for atmospheric light estimation. Zhu et al. have recently proposed new technique to compute transmission map using energy minimization and refinement of transmission map is done using piece-wise smoothing filter [18]. The benefit of this algorithm is to generate artifacts free defogged image. In [19], globally guided image filter-based algorithm is proposed which helps to improve color contrast of defogged image. Ju et al. have proposed improved atmospheric scattering model and the resultant image contains halo artifacts, in addition to fog factor and discoloration [20]. Lee et al. proposed a novel approach by combining defogging algorithm with de-mosaicking algorithm simultaneously [21]. The combined algorithm has benefits of removing noise in distant scenes. In [22], an algorithm based on weighted least square and high dynamic range is proposed which is used to preserve edge information in RGB images. In [23], Majeed et al. have used hybrid median filter and accelerated local laplacian filter for image Dehazing. Visual enhancement and color restoration is done using lo based gradient image decomposition. Tufail et al. have proposed an improved DCP based on RGB and YCbCr color spaces where transmission map refinement is done using Laplacian and mean filters [24]. As a result of refined DCP, better structural detail and enhanced color range is achieved. Makarau et al. have proposed DCP and bright channel prior (BCP) based on local search and image segmentation to effectively remove fog [25]. Makarau et al. have extended their work in [26] to improve dehazing technique by finding haze thickness using visible and cirrus bands.

In [27], image segmentation algorithm is proposed by Liu et al. in which RGB image is converted to YUV color space followed by image segmentation using region growing technique by selecting seed points. Seed points are selected on the basis of different fog density regions and atmospheric light is estimated using quad-tree hierarchical search algorithm. Similarly, in [28], Hong and Cai have proposed segmentation-based defogging algorithm in which they have divided image into sky and non-sky segments using Otsu segmentation technique. Dark channel prior is computed using adaptive parameter and image fusion is applied to combine defogged sky and non-sky segments. However, resultant images contain color distortion and seem unnatural. In another work, Cai et al. have proposed segmentation-based defogging algorithm using adaptive factor [29]. In this algorithm, segmentation is done using binary thresholding to compute sky and non-sky regions. After defogging, fusion is applied to combine sky and non-sky regions. The color scheme is distorted at the edges of resultant image. In [30], reliability map of DCP-based algorithm is proposed. In this technique, they have found out the reliable regions which contain sufficient prior knowledge to compute DCP. Transmission map is estimated using the reliable pixels only. Linear fitting curve is then used to find the transmission map for unreliable pixel values. Qing et al. have proposed K-means clustering for depth maps calculation to defog input image [31]. K-means is used to segment image into multiple parts having different depth range; then, transmission map is computed for each region. the edges of resultant image are not smooth, and the fog factor is also available at the edges.

The conditional Generative Adversarial Network is proposed in [32] which directly removes haze from an image, without estimating transmission map. Image defogging is performed by an end-to-end trainable neural network. Some other machine learning-based algorithms are also proposed in [33–40]. Choi et al. have proposed fog-aware density evaluator (FADE) and density-based referenceless perceptual defogger (DEFADE) in [33]. In [34], Li et al. have proposed residual-based deep CNN dehazing algorithm. Work consist of transmission map estimation network and dehazing using residual network. The advantage of residual-based network is its reduced computational complexity as it does not require atmospheric light estimation. Existing state-of-the-art algorithms fail to completely remove fog from foggy image and the results are either over saturated or low contrast. So, to avoid the over saturation and better remove the fog particles, a modified DCP-based algorithm is proposed. The major contributions of this work are as follows:

• 1. Modified dark channel prior is computed.

• 2. Transmission map based on fog density is estimated.

• 3. Segmentation-based defogging algorithm is proposed

• 4. We tested our algorithm using RESIDE images dataset which includes both natural and synthetic images. The results have proven that our algorithm achieves the state-of-the-art performance on the proposed defogging algorithm.

2 Proposed methodology

We have proposed two approaches for image defogging task. The first one utilizes the modified dark channel prior for image defogging, and the second one incorporates segmentation of the input image for localized parameter estimation before applying defogging algorithm.

2.1 Image defogging using modified dark channel prior

In this work, a modified dark channel is computed for transmission map estimation. Later, transmission map is refined using guided image filter (GIF). GIF is more efficient than the other refinement filters as it reduces the overall computation time for the transmission map refinement, hence optimizing the defogging algorithm. The work flow of our proposed technique is shown in Fig. 2, which illustrates that the proposed model is applied to foggy image to compute dark channel and atmospheric light. Atmospheric light is used to estimate transmission map which is refined in order to preserve the gradient information. The refined transmission map is then used to generate fog-free image.

Fig. 2

DCP-based algorithm for image defogging

2.1.1 Dark channel and atmospheric light estimation

First, a dark channel is computed to estimate atmospheric light using Eq. (1), that is obtained from [11]. A minimum filter of window size ω is applied to compute dark channel where ω is kept as 31×31. Foggy images along with the corresponding dark channels are shown in Fig. 3.

$$ I_{\text{DCP}} =\min_{x \in w(k)}(\min_{y\in \{R,G,B\}}I(y)) $$

(1)

Fig. 3

Dark channel computation. a Foggy images b Corresponding dark channels

After successfully computing dark channel, atmospheric light A_light is estimated which is a 3×1 vector containing highest intensity values that are computed from 0.1% brightest pixels of dark channel. Algorithm 1 illustrates the pseudo code to compute atmospheric light where, I is input foggy image, I_DCP is dark channel prior, N is 0.1% brightest pixel values In the dark channel and m is temporary variable. xxy is the size of dark channel prior. As a first step, I_DCP is compared with max value of N. If both values are equivalent; then. input image pixels corresponding to N pixel locations are considered as the atmospheric light.

2.1.2 Transmission map estimation and refinement

Atmospheric light is used to compute transmission map. A transmission map as given in (2) is computed for each RGB color channel by dividing input image with its corresponding color channel atmospheric light.

$$ T(x) = 1-\zeta [\frac{I(x)}{A}] $$

(2)

Here, value of ζ depends upon fog density of the hazy input image. Fog density (FD) is computed using FADE [14]. where, f_FD1 and f_FD2 are fixed throughout the algorithm and the values are chosen to avoid oversaturation and to completely remove fog from input image. f_FD1 and f_FD2 are set to 2.5 and 1, respectively. These values are achieved after performing cross validation on RESIDE dataset to estimate the accuracy of the performance of a proposed model.

Algorithm 2 illustrates the pseudo code for estimation and refinement of transmission map. After computing transmission map, refinement of transmission map is required to preserve gradient information. In our proposed methodology, guided image filter (GIF) is used for the refinement process where input image itself is used as guidance image as edge preserving and smoothing filter. GIF is faster than other refinement filters as it reduces the overall computational complexity of the defogging algorithm [41]. Therefore,

$$ T_{\text{refined}}(x) = a_{k} T_{k} + b_{k} \quad \forall k \in W_{k} $$

(3)

where, T_refined is a linear transform of T in a window of size W. a and b are linear coefficients that are constant in W_k. Transmission map and refined transmission maps are shown in Figs. 4 and 5, respectively. After refining the transmission map, defogged image is reconstructed using:

$$ R(x) = \frac{I(x)-A}{T_{\text{refined}}+ \epsilon}+A $$

(4)

Fig. 4

Transmission map estimation. a Foggy images b Transmission maps

Fig. 5

Transmission map refinement. a Foggy images. b Refined transmission maps

Where, ε is a constant with negligibly small value to avoid division with zero. At the end of defogging algorithm, gamma correction is also used to improve the overall brightness of reconstructed image.

2.2 Segmentation-based image defogging using modified dark channel prior

In this work, we have proposed defogging algorithm using image segmentation technique. Image segmentation is done using graph-based segmentation technique. The segments are based on sky and non-sky regions. Dark channel and atmospheric light is computed for each segment. Transmission map is estimated on the basis of average value of atmospheric light. The refinement process is same as discussed in previous methodology, that is, based on guided image filter. Segmentation-based algorithm is efficient in terms of removing fog particles, yield high SSIM and PSNR, and lower value of MSE. Segmentation-based proposed algorithm is shown in Fig. 6.

Fig. 6

Segmentation-based image defogging using modified dark channel prior

In this approach, a semi-automatic segmentation is used to convert foggy image into sky and non-sky segments. Foreground and background pixels are selected manually to convert image into two segments. Scribbles are drawn onto the image, which divides the image into background and foreground pixels and then graph theory is applied for fast segmentation. Resultant segmented images are shown in Fig. 7. After converting foggy image into sky and non-sky segments, dark channel is computed for each segment using Eq. (5). A minimum filter of window size ω is applied to compute dark channel, where ω is kept at 31×31 for optimized results. Foggy images along with dark channel are shown in Fig. 8.

$$ I_{\text{DCP}}(\text{seg}_{i}) = \min_{x \in w\left(k\right)}\min_{y\in \left\{R,G,B\right\}}I_{\text{seg}_{i}}(Y) \quad i = 1,2 $$

(5)

Fig. 7

Image segmentation using graph cut theory. a Foggy image. b Sky regions. c Non-sky regions

Fig. 8

Dark channels for segmented images. a Sky and non-sky regions of foggy image. b Corresponding dark channels for each segment

After successfully computing dark channels, atmospheric light is estimated using each dark channel. Final atmospheric light is computed using the average values of each atmospheric light.Transmission map estimation and refinement is done using the same procedure as discussed in first proposed methodology. The resultant transmission map along with refined transmission map computed using the average value of atmospheric light are shown in Fig. 9. After refining the transmission map, defogged image is reconstructed. Fog-free image is reconstructed using Eq. (4).

Fig. 9

Transmission maps using image segmentation a Foggy image. b Transmission map. c Refined transmission map

3 Results and discussion

Experiments are performed using realistic single-image dehazing (RESIDE) [1] dataset which contains outdoor hazy images including natural, as well as, synthetically generated foggy images. Dataset is divided into four sub-categories that are shown in Table 1.

Table 1 Overview of RESIDE data source and content

The experimental results are compared with He [11], Zahid [24], DEFADE [33], and Tarel [42]. Image quality, visibility enhancement, edge and texture information, color, and structure of image are important factors to evaluate defogging algorithm [43]. For evaluation of proposed algorithm results, full reference metric such as structural similarity index metric (SSIM) [44], peak signal to noise ratio (PSNR), full-reference image quality assessment (FR-IQA) [45], and mean square error (MSE) are compared with current state of the art algorithms. Other than these metrics, some non-reference metrics including naturalness image quality evaluator (NIQE) [46], fog density, blind/referenceless image spatial quality evaluator (BRISQUE) [47], and dehazing algorithms index (DHQI) [48] are also calculated.

PSNR is fully reference based metric that requires ground truth image. PSNR is computed by:

$$ \text{PSNR} = 10 \log_{10} \frac{\text{MAX}^{2}}{\text{MSE}} $$

(6)

Where MAX represents highest possible pixel value in an input image. MSE is mean square error, given by:

$$ \text{MSE} = \frac{1}{jk} \sum_{i_{r} = 0}^{j-1} \sum_{i_{c} = 0}^{k - 1} \left[I_{\text{input}} (i_{r},i_{c}) - I_{\text{output}} (i_{r},i_{c})\right]^{2} $$

(7)

where, I_input and I_output are input and fog-free reconstructed output image, respectively, and j×k is size of the image. If the PSNR is larger, image distortion will be smaller. So, for better performance of dehazing algorithm, PSNR must be higher. SSIM also requires reference image and it compares similarity of resultant image with ground truth image depending upon brightness, contrast, and structure. On the other hand, range of SSIM score is 0 to 1 and value closer to 1 is more appealing to the researchers. SSIM is defined as:

$$ \text{SSIM} = \frac{\left(2 \mu_{x_{in}}\mu_{y_{\text{out}}} + k_{1})(2 \sigma_{x_{in}y_{\text{out}}}+k_{2}\right) }{\left(\mu_{x_{\text{in}}}^{2} + \mu_{y_{\text{out}}}^{2} + k_{1} \right)\left(\sigma_{x_{\text{in}}}^{2} + \sigma_{y_{\text{out}}}^{2} + k_{2}\right)} $$

(8)

where x_in and y_out are two windows of common size, $\mu _{x_{\text {in}}}$ and $\mu _{y_{\text {out}}}$ are mean of windows, $\sigma _{x_{\text {in}}}^{2}$ and $\sigma _{y_{\text {out}}}^{2}$ are their variances and k₁ and k₂ are constants. Results generated using modified DCP-based algorithm and image segmentation-based algorithms are shown in Fig. 10. Figures 11 and 12 present a comparison of proposed algorithm with different popular algorithms using synthetically generated foggy dataset and natural foggy dataset, respectively. Figure 11 illustrates poor color contrast and darker effects in He et al. [11] results. Also the DEFADE [33] results show higher fog factor as shown. Proposed algorithms results, comparatively, seem better as compared to the other algorithms.

Fig. 10

Proposed methodology results. a Input images. b Ground truth images. c Defogged images using modified DCP-based algorithm. d Defogged images using segmentation-based algorithm

Fig. 11

Comparison of different defogging algorithms using synthetically generated hazy dataset. a Input images. b Ground truth images. c Tarel [42]. d He [11]. e DEFADE [33]. f Zahid [24]. g Modified DCP-based proposed algorithm. h Segmentation-based proposed algorithm

Fig. 12

Comparison of different defogging algorithms using real hazy dataset. a Input images. b Ground truth images. c Tarel [42]. d He [11]. e DEFADE [33]. f Zahid [24]. g Modified DCP-based proposed algorithm. h Segmentation-based proposed algorithm

Quantitative comparison is presented in Table 2. The proposed algorithm generates result with highest SSIM, DHQI, FR-IQA and PSNR and lowest Fog density and MSE. The experiments are performed on hundred different images. Figure 13 depicts the complete comparison of proposed algorithm with existing state of the art algorithms through box plot. In each box, median is indicated by central red mark, the top and bottom edges of box indicate first and third quartile, respectively. The whiskers above and below the box show the maximum and minimum values, respectively. “+” symbol indicates outliers. Figure 13 shows that the image quality of defogged image generated using proposed algorithm is better than other algorithms. It further shows that PSNR is highest for proposed algorithm and lowest for Tarel [42]. The standard deviation of PSNR is highest for DEFADE [33]. Comparison of SSIM, non-reference DHQI and full-reference IQA is also presented in Fig. 13 which shows the better performance of proposed algorithm. It also presents fog density and MSE comparison for five different state of the art algorithms. Our results contain lower value of MSE and NIQE which have proven promising results.

Fig. 13

Box plot for average quantitative comparison of 100 images for different state-of-the-art defogging algorithms. a No-reference quality metric BRISQUE, b PSNR, c SSIM, d fog density, e MSE and f NIQE, g DHQI, and h FR-IQA

Table 2 Quantitative comparison of proposed algorithm with existing state-of-the-art algorithms

The experimental results are compared with current state-of-the-art machine learning algorithms including MSCNN [36], Dehaze-Net [37], and AoD-Net [38]. Figure wise comparison is presented in Fig. 14. It shows that the results generated using MSCNN [36] are oversaturated and contain low similarity index with ground truth image. It further shows that results generated using Dehaze-Net [37] still contains fog particles. Table 3 shows quantitative comparison of these algorithms which has proven that proposed algorithms either exceeds or comparable to the machine learning-based algorithms which require large amount of data for training and are generally computationally expensive.

Fig. 14

Comparison of proposed algorithm with current state of the art machine learning-based defogging algorithms. a Input image. b Ground truth image. c Dehaze-Net [37]. d MSCNN [36]. e Modified DCP-based proposed algorithm. f Segmentation-based proposed algorithm

Table 3 Quantitative comparison of proposed algorithm with existing state-of-the-art machine learning-based image defogging

4 Conclusion

In this paper, we have proposed two different techniques for image defogging. The first one consists of a modified DCP with guided image filter to estimate and refine the transmission map in order to preserve edge information. We also proposed a segmentation-based defogging in which the foggy image is divided into two segments based on sky and non-sky regions. The dark channel for each segment is computed using the modified DCP algorithm. Proposed algorithms could be selected depending upon application. For indoor and less sky-region-based outdoor applications, modified DCP-based algorithm is preferable. Segmentation-based algorithm generates significant results on outdoor images having sky and non-sky regions. The proposed algorithms are tested against multiple state-of-the-art approaches using the RESIDE dataset [1]. The quantitative and qualitative comparisons are performed for effective evaluation and validation. The results generated using the proposed algorithm show higher values of SSIM and PSNR. The comparative analysis proved that the results generated using the proposed algorithm are better than existing algorithms with balanced luminance and saturation with a lower mean square error and fog density factor.