HDR Image Quality Enhancement Based on Spatially Variant Retinal Response

Abstract

There is a growing demand for being able to display high dynamic range (HDR) images on low dynamic range (LDR) devices. Tone mapping is a process for enhancing HDR image quality on an LDR device by converting the tonal values of the original image from HDR to LDR. This paper proposes a new tone mapping algorithm for enhancing image quality by deriving a spatially-variant operator for imitating S-potential response in human retina, which efficiently improves local contrasts while conserving good global appearance. The proposed tone mapping operator is studied from a system construction point of view. It is found that the operator is regarded as a natural extension of the Retinex algorithm by adding a global adaptation process to the local adaptation. The feasibility of the proposed algorithm is examined in detail on experiments using standard HDR images and real HDR scene images, comparing with conventional tone mapping algorithms.

1. Introduction

From starlight to sunlight, light intensity in natural scenes of real world can have high dynamic range (HDR) with ten or higher order of magnitude. A certain level of HDR images is available due to recent advances in HDR camera technology. Also we can apply a multiple exposure technique to a low dynamic range (LDR) camera so that an HDR image is obtained from multiple LDR images. On the other hand, the dynamic range of a general display device is too narrow to accept the HDR image. Therefore, many tone mapping algorithms have been developed for transforming an HDR image effectively into an LDR image, so that appearance of the original scene can be reproduced on the LDR displays.

The recent literatures on HDR tone mapping algorithms are extensively reviewed in [1, 2]. Most of the tone mapping techniques can be classified into two broad categories, global and local operators. The tone mapping operators of the global technique reduce the dynamic range by a single appropriately designed spatially invariant mapping function [3–7]. Let be an image intensity at pixel captured by a camera. This is simply mapped to a modified image intensity, , where is a compressive function such as a power function, or a function that is adapted to the image histogram. However, the contrast of details is sacrificed. The converted images often look washed out because the same mapping function is applied to all pixels.

In contrast, the operators of the local technique use a mapping that varies spatially depending on the local pixel statistics and local pixel context [8–20]. The image intensity at pixel is simply considered as the product of surface reflectance and illuminant . When inferring the illuminant , we can restore the reflectance with from the captured image . To estimate the local distribution of illumination , an average within a local region of the image is computed in such several ways as the arithmetic average, the geometric average, and the Gaussian-blurred operation. However, in such a case the overall contrast is sacrificed. A recent approach is that the local tone mapping operator is adopted for the multiscale decomposition of an image on different scales [15–18]. Subband decomposition techniques including Laplacian pyramids, wavelets, and Gabor transforms were also proposed as a new approach to the tone mapping problem [19, 20]. However, those multiscale and subband techniques have a lot of arbitrary steps for synthesizing images and determining optimum parameters. So, those recent approaches are still a trial and error stage technologically.

A human being has an ideal tone reproduction system in the human visual system (HVS). HVS is capable of simultaneously perceiving light intensities over a range of 3 orders of magnitude, and with brightness adaptation, its sensitivity can stretch to 10 orders of magnitude. It is noted that the adaptation process especially plays an important role in visual appearance of any viewed scene [5, 10, 13, 14]. The present paper develops a new tone mapping technique based on the adaptation mechanism of HVS. Our tone mapping algorithm takes inspiration from the nonlinear adaptation that occurs in the retina, which efficiently improves local contrasts while conserving good global appearance. Especially our technique uses S-potential response in the retina [19, 20]. Although this response function was already applied to the tone mapping problem in a few literatures [10, 13, 14], most of the previous techniques used this response property as a spatially invariant operator. Actually, we see real-world scene while varying the S-potential response spatially.

In this paper a spatially variant operator is devised for imitating the S-potential function and realizing the local adaptation process such as brightness constancy in HVS. This operator is useful for both the global adaptation for an entire scene and the local adaptation around a gaze point within the scene. From a system construction point of view, it is meaningful to investigate a relationship of the proposed tone mapping operator to the traditional Retinex, which is the well-known local tone mapping operator. That is, the Retinex takes only the local adaptation into account. The proposed operator can be regarded as a natural extension of the Retinex by adding a global adaptation process to the local adaptation. From an image processing point of view, our operator has an essential advantage in computational simplicity and easy parameter setting based on physiological findings.

This paper is organized as follows. Section 2 develops a tone mapping algorithm based on HVS. Section 3 considers a system construction of the proposed tone mapping operator. We investigate a relationship to the Retinex algorithm. In Section 4, the performance of the proposed method is examined in detail on experiments using standard HDR images from a database and real HDR scene images from a calibrated imaging system.

2. Tone Mapping Algorithm Based on HVS

The overall impression of an entire image is reproduced by a global adaptation mechanism, and the local visibility is improved by changing adaptation levels according to local surround intensities of a gaze point.

2.1. Global Adaptation

When we look at a reproduced image on a display device or a printer output, we put the entire image in view. Therefore, a basic image reproduction process is the global tone mapping to the entire scene. A key mechanism of HVS for the global tone mapping is the mechanism of mediating adaptation to lighting conditions. We especially employ a model of photoreceptor adaptation that can describe a receptors' automatic adjustment to the general level of illumination. Compared to the broad range of background light intensities over which the human visual system works, the photoreceptors respond linearly to a rather narrow range of intensities. This linear range is only about 3 log units. The HVS adaptation process dynamically adjusts the narrow response function so that the response conforms better to the available light source.

Naka and Rushton [21] first proposed a model to describe the S-potentials in retina of fish. Dowling [22] also proposed a slightly modified model. They measured directly cellular response functions for cone, rod, and bipolar cells, and firing rates for sustained ON-center retinal ganglia. The measurements are then described properly as

(1)

Here, is the photoreceptor response, is the maximum response, is light intensity, and is an adaptation level. The quantity is generally called a semisaturation constant that represents the adaptation level with the condition of . The parameter is a sensitivity control exponent that has a value generally between 0.7 and 1.0 [22].

Figure 1 shows the responses of retina to the luminance intensities with different adaptation levels, which helps us to understand the adaptation process well. From left to right, the curves represent the responses of retina at adaptation levels 0.001, 0.01, 0.1, 1, 10, 100, and 1000, respectively. From Figure 1, we can see that the retinal response is an S-shape curve when the luminance intensities are drawn in logarithmic domain. This confirms that the HVS compresses very bright area and very dark shadow area in a scene dramatically, while keeping the middle range invariant to preserve well contrast.

Figure 1

Response of retina as a function of luminance value at different adaptation levels of σ = 10 ^-3 , 10 ^-2 , 10 ^-1 , 1, 10, 10 ² , and 10 ³ .

2.2. Operation of Local Adaptation

The local adaptation property in HVS is realized by slightly changing the adaptation level depending on the surrounding light intensity. Let us define to be a spatially variant value and a small change in . Let  [cd/m²] be an intensity of -channel at pixel in an HDR image. When we extend the model of (1), the LDR image output is derived as follows:

(2)

where, means the maximum value of the LDR output. Taking account of n close to 1, we can rewrite the above expression as

(3)

where indicates a displacement from the global adaptation level. We determine the displacement value of as difference in the image intensity between the center of a gaze point and its surround. Let and be the center image intensity of channel at pixel and the surrounding intensity, respectively. Then the displacement value is

(4)

In the case () that the local luminance is brighter than the center luminance, the corresponding S-shaped curve decided by the global tone-mapping moves to the right in Figure 2 for moving the saturated bright areas to a moderate linear contrast range. In the case () that the local distribution of illumination is darker than the center luminance, the S-shaped curve moves to the left in Figure 2 for moving the darkened areas to the linear contrast range. We consider adaptive global tone mapping with an automatic change of the adaptation level depending on the surrounding luminance level.

Figure 2

Local adaptation on the S-potential response curve.

Substituting (4) into (3) derives the following formula:

(5)

There may be various ways for computing the surrounding intensity . Durand and Dorsey introduced the bilateral filter for estimating the illumination distribution [11]. Bilateral filtering was developed by Tomasi and Manduchi as an alternative to anisotropic diffusion [23]. It is known as an edge-preserving smoothing operator that effectively blurs an image but keeps sharp edges intact. However, we note that a normal algorithm of the bilateral filter often causes the halo artifacts for HDR images as shown in Section 4. Here a multiple bilateral filter is proposed as an improved bilateral filter for reducing the haloing artifacts more significantly.

The proposed tone mapping operator is summarized by the following equations:

(6)

(7)

(8)

where is the standard deviation for a Gaussian f in the spatial domain such as

(9)

and is the standard deviation for a Gaussian g in the luminance domain. In our algorithm, multiple Gaussians are used in the luminance domain. Here, is a normalization factor and is the whole image. In (6), for 8-bit output device. can be derived by the multiple bilateral filtering of .

2.3. User Parameters

Since the algorithm has several user parameters, it is desired to easily determine the optimal parameters in actual tone reproduction applications. The present algorithm includes four kinds of parameters of , , , and , which control contrast, luminance, and edge preservation by the multiple bilateral filtering parameters, respectively. The sensitivity parameter was discussed in the literature [22], where n= 0.7 for long test flashes (seconds) and 1.0 for short test flashes (10 ms). In our operator, was better for most HDR images, because the exposure time for capturing HDR images is usually milliseconds. The semisaturation parameter means the global adaptation level in an HDR scene. In generally, the arithmetic average, the geometric average, or a Gaussian blurred version within a local region of the image can be used for determining this parameter. Our experiments to various HDR images suggest the superiority of the arithmetic average of the entire image. Therefore, we determine the global adaptation level automatically from the average intensity of an HDR image.

Parameters and are standard deviations of the multiple bilateral filtering. They use a Gaussian for f in the spatial domain and a Gaussian for in the intensity domain. Therefore, the value at a pixel is influenced mainly by neighboring pixels with a similar intensity. We note that an appropriate value of the Gaussian standard deviation for the spatial domain depends on the visual angle, because the parameter is used to remove the influence of the local illumination. Our experimental results suggest that 2-degree viewing angle was appropriate for test images. In the normal Gaussian distribution, 2 times of the standard deviation from the mean account for about 95%. Therefore, we set to 1/4 of pixels existing within 2-degree viewing angle as shown in Figure 3. If the pixel density within 2-degree viewing angle is unknown, it is appropriate experimentally to set to a small value. An appropriate value of for the intensity domain depends on the intensity range [cd/m²] of an HDR image. It was found empirically that two Gaussian functions with and for the multiple bilateral filter in (8) were appropriate for natural scenes with moderate intensity range.

Figure 3

Setting of Gaussian standard deviation σ _m .

3. Consideration on System Construction

Let us consider a system construction of the proposed tone mapping operator. The Retinex algorithm is well known for local tone mapping [24–26]. It can be meaningful to investigate a relationship to this algorithm. The local tone mapping problem is closely related to the problem of recovering reflectance from an image. An image intensity at each pixel point s is represented as a product of the reflectance R _s and the illuminance

(10)

The function is invariant on illumination and often referred to as the intrinsic image of a scene. A local tone mapping operation, in principle, is achieved by separating an image to the and components. Tumblin and Rushmeier used this approach for displaying high-contrast synthetic images [3], where the material properties of the surfaces and the illumination are known at each point in the image, making it possible to compute a perfect separation of an image to layers of lightning and surface properties. Rahman et al. presented a dynamic range compression method [9] based on a multiscale version of the Retinex theory for color vision. The Retinex estimates the reflectance as the ratio of to its low-pass filtration output. A similar operator was explored by Chiu et al. [8] and was also found to suffer from halo artifacts and compute the logarithm of the Retinex responses for several low-pass filters of different sizes and linearly combine the results.

In the above framework of the Retinex algorithm, the LDR output of -channel at pixel s is derived as follows:

(11)

where means the maximum value of the LDR output. Comparing the Retinex in (11) with the proposed algorithm in (5), the proposed algorithm can be regarded as a natural extension of the single-scale Retinex by adding an offset for the global adaptation to the local illuminance . It should be noted from an image processing point of view that the proposed algorithm is simple in computation and requires no additional steps such as multiscale and subband techniques for improving the Retinex. Furthermore, because the present algorithm is derived from a physiological model of HVS, the meanings of tone mapping parameters are clear as in Section 2.3.

Figure 4 depicts a system construction for the proposed tone mapping operation. The feature of our operational system is an adaptive global tone mapping with automatically changing the adaptation level that depends on the surrounding luminance level. This procedure is realized in the dashed box in Figure 4. In the process, input RGB intensities are divided by sum of the local illumination and the global illumination in the input luminance component. We note that if we remove the component of global illumination, the system reduces to the single-scale Retinex. Even in that case, the present system of estimating local luminance by the multiple bilateral filtering is more effective than the traditional Retinex.

Figure 4

System construction of the tone mapping operation.

4. Experiments

First, we apply the proposed algorithm to several standard HDR images and compare the results with the conventional algorithms. Second, we construct a calibrated imaging system for capturing HDR images and render LDR images by the proposed algorithm on a display device and printouts. The performance is examined on visual experiments in detail.

4.1. Evaluation on Standard HDR Images

Various HDR images are available from Mark's HDR photographic survey [27] and HDR DVD in the literature [2]. Here we use two images of "UR Chapel()" and "Las Vegas Store" from [27]. The parameter value of was used for these images because the actual pixel density was not available.

In the first chapel image, the image size is , and the luminance range exceeds the ratio of 1 : 10⁶. Figure 5 shows a set of the resulting tone-mapped images. Figure 5(a) is the result by a linear tone mapping. We can see only the bright part of stained glasses. Figures 5(b) and 5(c) are the results from the spatially invariant global operator by (1) and the local operator "Retinex" by (11), respectively. When looking at Figure 5(b), the overall impression looks realistic. However in the close-up view of Figure 5(f), we note that the brightness is saturated and the color is washed out. In contrast, the Retinex in Figure 5(c) may reproduce the local region clearly. However, the overall impression looks unnatural. We note that artifact appears around large step edges in Figure 5(g). Figures 5(d) and 5(g) are reproduced by the proposed operator. The resulting images have the advantage of both global and local points of view.

Figure 5

Reproduced images from the HDR image "UR Chapel(1)."Linear Global operator Local operator Proposed operator Close-up of (a)Close-up of (b)Close-up of (c)Close-up of (d)

In the second Las Vegas image, the image size is , and the luminance range exceeds 1 : 10⁶. Next, we verify the effectiveness of the multiple bilateral filter in (7) using "Las Vegas Store." Figure 6 shows a set of the tone-mapped images. Figure 6(a) is the result by the spatially invariant global operator (1). Figures 6(b)–6(d) show the tone-mapped images with different values of σ _d by our spatially variant operator, where Figures 6(b) and 6(c) are the results by a single bilateral filter with and 0.1, respectively, and Figure 6(d) is the result by the multiple bilateral filter with . As shown in Figure 6(a), the entire reproduction image has become unclear using the spatially invariant global operator. For instance, the character string of "Welcome to Falrilous" cannot be recognized in the close-up image. As shown in Figure 6(b), characters are still not well reproduced. In Figure 6(c), unnatural halos appear around lights, although the characters are well reproduced. Such problems as character reproduction and halo artifacts can be solved in Figure 6(d) by the proposed algorithm.

Figure 6

Reproduced images from the HDR image "Las Vegas Store. "Spatially invariant global operator Spatially variant operator with single bilateral filter (σ_d = 0.01) Spatially variant operator with single bilateral filter (σ_d = 0.03) Proposed operator with a multiple bilateral filter ({σ_d} = {0.01, 0.3})

4.2. Evaluation on a Calibrated Imaging System

A calibrated imaging system using a Canon camera with 12 bit depth was constructed for capturing real world scenes. We confirmed linearity of the camera output by using gray scale patches of X-Rite ColorChecker. Multiple images of the same scene captured with different exposure times were then combined into an extended range image with 16 bits. The basic computational procedure is shown in [28]. For example, Figure 7 shows a set of the captured images with five different exposure times from 1/8 to 1/125 (sec) for the HDR scene of "Meeting Room."

Figure 7

Camera images captured with different exposure times for the scene "Meeting Room."1/8 (sec)1/15 (sec)1/30 (sec)1/60 (sec)1/125 (sec)

The proposed algorithm was applied to HDR images obtained by the present imaging system. As shown in Figure 8, an HDR "Desk" scene including the X-Rite ColorChecker on a table was photographed under an incandescent light bulb of 500 W in the upper right direction within a darkroom. Since the field angle of the real scene was beyond 2 degrees, we could not observe the whole scene simultaneously. Moreover, another HDR scene "Meeting Room" was captured in an indoor meeting room at Chiba University as shown in Figure 9. We note that the scene contains the ColorChecker on a chair, which is illuminated by the three types of light source of outside daylight, fluorescent ceiling lamps in the room, and fluorescent ceiling lamps in the passage. In all experiments, we used the same user parameters as in Section 2.3.

Figure 8

Reproduced images from "Desk."Proposed operatorSchlick [7]Rahman et al. [9]Durand and Dorsey [11]Reinhard et al. [12]Wang et al. [13]Pattanaik et al. [16]Li et al. [20]

Figure 9

Reproduced images from "Meeting room."Proposed operatorSchlick [7]Rahman et al. [9]Durand and Dorsey [11]Reinhard et al. [12]Wang et al. [13]Pattanaik et al. [16]Li et al. [20]

For performance comparison of the proposed algorithm with the other tone mapping algorithms, we selected seven famous algorithms by Schlick [7], Rahman et al. [9], Durand and Dorsey [11], Reinhard et al. [12], Wang et al. [13], Pattanaik et al. [16], and Li et al. [20]. The mapping results are shown in Figures 8 and 9. These algorithms have many parameters. The parameters in each algorithm were adjusted so that appearance of the resulting images was close to real scenes.

Visual experiments were performed based on evaluations viewing the real scenes [29]. We used two devices of a display and a printout for checking the device-dependency in the evaluation results. For this purpose, the tone-mapped LDR image of "Desk" was reproduced on an Eizo LCD monitor with the Adobe RGB color gamut, and the image of "Meeting room" was reproduced on a glossy paper by an Epson inkjet printer.

We have conducted a ranking-based subjective perceptual experiment with reference real scenes. Figure 10 shows the experimental scenes for the subjective evaluation. We presented simultaneously the real-world HDR scene and the tone-mapped images of each scene to human observers. The task of each subject was to arrange the eight reproduced images in order of score 1–8 according to the overall image quality, where 8 represents the best quality and 1 is the worst. Moreover, subjects were asked to describe what they paid attention to during the evaluation procedure. Ten subjects participated in the experiments. In the case of "Desk," subjects just observed the reproduced images on the monitor in a darkroom as shown in Figure 10(a). As shown in Figure 10(b), the printouts were observed in the same meeting room under the same standard illumination.

Figure 10

Environments for subjective perceptual evaluation. DeskMeeting Room

All experimental results are summarized in Figure 11. The bar chart indicates the average score of both experiments, and each straight line on the bar indicates the standard deviation of all scores. It should be noted that the proposed algorithm obtains a remarkable score with a high average and a small standard deviation. The Li algorithm has the highest score for only "Desk," and the Pattanaik algorithm has a very high score for only "Meeting Room." However, the evaluations of these algorithms are relatively lower in the other scenes so that their standard deviations become large. In contrast the proposed algorithm is evaluated stably and highly. It gives visually pleasing LDR images and is successful in making details visible in both bright and dark regions without any artifacts.

Figure 11

Evaluation results.

5. Conclusions

We have proposed a novel tone mapping algorithm for effectively reproducing HDR images on devices with limited dynamic range of intensity. We incorporated the mechanism of global adaptation and local adaptation in the algorithm to imitate brightness constancy in HVS. The overall impression of an entire image was reproduced by the global adaptation mechanism, and the local visibility of an image was improved by changing the adaptation levels according to local surrounding intensities of a gaze point within the scene. The proposed tone mapping operator was studied from a system construction point of view. Then we found that our operator could be regarded as a natural extension of the Retinex algorithm by adding a global adaptation process to the local adaptation. The feasibility of the proposed method was verified on experiments using standard HDR images and real HDR scene images comparing with conventional tone mapping operators. As a next stage, the authors will study a color perceptual model for the tone mapping operator in the future.