HDR Image Quality Enhancement Based on Spatially Variant Retinal Response
© Takahiko Horiuchi and Shoji Tominaga. 2010
Received: 10 May 2010
Accepted: 23 September 2010
Published: 27 September 2010
There is a growing demand for being able to display high dynamic range (HDR) images on low dynamic range (LDR) devices. Tone mapping is a process for enhancing HDR image quality on an LDR device by converting the tonal values of the original image from HDR to LDR. This paper proposes a new tone mapping algorithm for enhancing image quality by deriving a spatially-variant operator for imitating S-potential response in human retina, which efficiently improves local contrasts while conserving good global appearance. The proposed tone mapping operator is studied from a system construction point of view. It is found that the operator is regarded as a natural extension of the Retinex algorithm by adding a global adaptation process to the local adaptation. The feasibility of the proposed algorithm is examined in detail on experiments using standard HDR images and real HDR scene images, comparing with conventional tone mapping algorithms.
From starlight to sunlight, light intensity in natural scenes of real world can have high dynamic range (HDR) with ten or higher order of magnitude. A certain level of HDR images is available due to recent advances in HDR camera technology. Also we can apply a multiple exposure technique to a low dynamic range (LDR) camera so that an HDR image is obtained from multiple LDR images. On the other hand, the dynamic range of a general display device is too narrow to accept the HDR image. Therefore, many tone mapping algorithms have been developed for transforming an HDR image effectively into an LDR image, so that appearance of the original scene can be reproduced on the LDR displays.
The recent literatures on HDR tone mapping algorithms are extensively reviewed in [1, 2]. Most of the tone mapping techniques can be classified into two broad categories, global and local operators. The tone mapping operators of the global technique reduce the dynamic range by a single appropriately designed spatially invariant mapping function [3–7]. Let be an image intensity at pixel captured by a camera. This is simply mapped to a modified image intensity, , where is a compressive function such as a power function, or a function that is adapted to the image histogram. However, the contrast of details is sacrificed. The converted images often look washed out because the same mapping function is applied to all pixels.
In contrast, the operators of the local technique use a mapping that varies spatially depending on the local pixel statistics and local pixel context [8–20]. The image intensity at pixel is simply considered as the product of surface reflectance and illuminant . When inferring the illuminant , we can restore the reflectance with from the captured image . To estimate the local distribution of illumination , an average within a local region of the image is computed in such several ways as the arithmetic average, the geometric average, and the Gaussian-blurred operation. However, in such a case the overall contrast is sacrificed. A recent approach is that the local tone mapping operator is adopted for the multiscale decomposition of an image on different scales [15–18]. Subband decomposition techniques including Laplacian pyramids, wavelets, and Gabor transforms were also proposed as a new approach to the tone mapping problem [19, 20]. However, those multiscale and subband techniques have a lot of arbitrary steps for synthesizing images and determining optimum parameters. So, those recent approaches are still a trial and error stage technologically.
A human being has an ideal tone reproduction system in the human visual system (HVS). HVS is capable of simultaneously perceiving light intensities over a range of 3 orders of magnitude, and with brightness adaptation, its sensitivity can stretch to 10 orders of magnitude. It is noted that the adaptation process especially plays an important role in visual appearance of any viewed scene [5, 10, 13, 14]. The present paper develops a new tone mapping technique based on the adaptation mechanism of HVS. Our tone mapping algorithm takes inspiration from the nonlinear adaptation that occurs in the retina, which efficiently improves local contrasts while conserving good global appearance. Especially our technique uses S-potential response in the retina [19, 20]. Although this response function was already applied to the tone mapping problem in a few literatures [10, 13, 14], most of the previous techniques used this response property as a spatially invariant operator. Actually, we see real-world scene while varying the S-potential response spatially.
In this paper a spatially variant operator is devised for imitating the S-potential function and realizing the local adaptation process such as brightness constancy in HVS. This operator is useful for both the global adaptation for an entire scene and the local adaptation around a gaze point within the scene. From a system construction point of view, it is meaningful to investigate a relationship of the proposed tone mapping operator to the traditional Retinex, which is the well-known local tone mapping operator. That is, the Retinex takes only the local adaptation into account. The proposed operator can be regarded as a natural extension of the Retinex by adding a global adaptation process to the local adaptation. From an image processing point of view, our operator has an essential advantage in computational simplicity and easy parameter setting based on physiological findings.
This paper is organized as follows. Section 2 develops a tone mapping algorithm based on HVS. Section 3 considers a system construction of the proposed tone mapping operator. We investigate a relationship to the Retinex algorithm. In Section 4, the performance of the proposed method is examined in detail on experiments using standard HDR images from a database and real HDR scene images from a calibrated imaging system.
2. Tone Mapping Algorithm Based on HVS
The overall impression of an entire image is reproduced by a global adaptation mechanism, and the local visibility is improved by changing adaptation levels according to local surround intensities of a gaze point.
2.1. Global Adaptation
When we look at a reproduced image on a display device or a printer output, we put the entire image in view. Therefore, a basic image reproduction process is the global tone mapping to the entire scene. A key mechanism of HVS for the global tone mapping is the mechanism of mediating adaptation to lighting conditions. We especially employ a model of photoreceptor adaptation that can describe a receptors' automatic adjustment to the general level of illumination. Compared to the broad range of background light intensities over which the human visual system works, the photoreceptors respond linearly to a rather narrow range of intensities. This linear range is only about 3 log units. The HVS adaptation process dynamically adjusts the narrow response function so that the response conforms better to the available light source.
Here, is the photoreceptor response, is the maximum response, is light intensity, and is an adaptation level. The quantity is generally called a semisaturation constant that represents the adaptation level with the condition of . The parameter is a sensitivity control exponent that has a value generally between 0.7 and 1.0 .
2.2. Operation of Local Adaptation
There may be various ways for computing the surrounding intensity . Durand and Dorsey introduced the bilateral filter for estimating the illumination distribution . Bilateral filtering was developed by Tomasi and Manduchi as an alternative to anisotropic diffusion . It is known as an edge-preserving smoothing operator that effectively blurs an image but keeps sharp edges intact. However, we note that a normal algorithm of the bilateral filter often causes the halo artifacts for HDR images as shown in Section 4. Here a multiple bilateral filter is proposed as an improved bilateral filter for reducing the haloing artifacts more significantly.
and is the standard deviation for a Gaussian g in the luminance domain. In our algorithm, multiple Gaussians are used in the luminance domain. Here, is a normalization factor and is the whole image. In (6), for 8-bit output device. can be derived by the multiple bilateral filtering of .
2.3. User Parameters
Since the algorithm has several user parameters, it is desired to easily determine the optimal parameters in actual tone reproduction applications. The present algorithm includes four kinds of parameters of , , , and , which control contrast, luminance, and edge preservation by the multiple bilateral filtering parameters, respectively. The sensitivity parameter was discussed in the literature , where n= 0.7 for long test flashes (seconds) and 1.0 for short test flashes (10 ms). In our operator, was better for most HDR images, because the exposure time for capturing HDR images is usually milliseconds. The semisaturation parameter means the global adaptation level in an HDR scene. In generally, the arithmetic average, the geometric average, or a Gaussian blurred version within a local region of the image can be used for determining this parameter. Our experiments to various HDR images suggest the superiority of the arithmetic average of the entire image. Therefore, we determine the global adaptation level automatically from the average intensity of an HDR image.
3. Consideration on System Construction
The function is invariant on illumination and often referred to as the intrinsic image of a scene. A local tone mapping operation, in principle, is achieved by separating an image to the and components. Tumblin and Rushmeier used this approach for displaying high-contrast synthetic images , where the material properties of the surfaces and the illumination are known at each point in the image, making it possible to compute a perfect separation of an image to layers of lightning and surface properties. Rahman et al. presented a dynamic range compression method  based on a multiscale version of the Retinex theory for color vision. The Retinex estimates the reflectance as the ratio of to its low-pass filtration output. A similar operator was explored by Chiu et al.  and was also found to suffer from halo artifacts and compute the logarithm of the Retinex responses for several low-pass filters of different sizes and linearly combine the results.
where means the maximum value of the LDR output. Comparing the Retinex in (11) with the proposed algorithm in (5), the proposed algorithm can be regarded as a natural extension of the single-scale Retinex by adding an offset for the global adaptation to the local illuminance . It should be noted from an image processing point of view that the proposed algorithm is simple in computation and requires no additional steps such as multiscale and subband techniques for improving the Retinex. Furthermore, because the present algorithm is derived from a physiological model of HVS, the meanings of tone mapping parameters are clear as in Section 2.3.
First, we apply the proposed algorithm to several standard HDR images and compare the results with the conventional algorithms. Second, we construct a calibrated imaging system for capturing HDR images and render LDR images by the proposed algorithm on a display device and printouts. The performance is examined on visual experiments in detail.
4.1. Evaluation on Standard HDR Images
Various HDR images are available from Mark's HDR photographic survey  and HDR DVD in the literature . Here we use two images of "UR Chapel( )" and "Las Vegas Store" from . The parameter value of was used for these images because the actual pixel density was not available.
4.2. Evaluation on a Calibrated Imaging System
For performance comparison of the proposed algorithm with the other tone mapping algorithms, we selected seven famous algorithms by Schlick , Rahman et al. , Durand and Dorsey , Reinhard et al. , Wang et al. , Pattanaik et al. , and Li et al. . The mapping results are shown in Figures 8 and 9. These algorithms have many parameters. The parameters in each algorithm were adjusted so that appearance of the resulting images was close to real scenes.
Visual experiments were performed based on evaluations viewing the real scenes . We used two devices of a display and a printout for checking the device-dependency in the evaluation results. For this purpose, the tone-mapped LDR image of "Desk" was reproduced on an Eizo LCD monitor with the Adobe RGB color gamut, and the image of "Meeting room" was reproduced on a glossy paper by an Epson inkjet printer.
We have proposed a novel tone mapping algorithm for effectively reproducing HDR images on devices with limited dynamic range of intensity. We incorporated the mechanism of global adaptation and local adaptation in the algorithm to imitate brightness constancy in HVS. The overall impression of an entire image was reproduced by the global adaptation mechanism, and the local visibility of an image was improved by changing the adaptation levels according to local surrounding intensities of a gaze point within the scene. The proposed tone mapping operator was studied from a system construction point of view. Then we found that our operator could be regarded as a natural extension of the Retinex algorithm by adding a global adaptation process to the local adaptation. The feasibility of the proposed method was verified on experiments using standard HDR images and real HDR scene images comparing with conventional tone mapping operators. As a next stage, the authors will study a color perceptual model for the tone mapping operator in the future.
The authors would like to thank Mr. Yuta Fukuda, Chiba University for his help in experiments.
- Devlin K, Chalmers A, Wilkey A, Purgathofer W: Tone reproduction and physically based spectral rendering. In State of The Art Reports. Eurographics; 2002:101-123.Google Scholar
- Reinhard E, Ward G, Pattanaik S, Debevec P: High Dynamic Range Imaging: Aquisition, Display, and Image-Based Lighting. Morgan Kaufmann, San Francisco, Calif, USA; 2005.Google Scholar
- Tumblin J, Rushmeier H: Tone reproduction for realistic images. IEEE Computer Graphics and Applications 1993,13(6):42-48. 10.1109/38.252554View ArticleGoogle Scholar
- Ward G: A contrast-based scalefactor for luminance display. Graphic Gems 1994, 415-421.View ArticleGoogle Scholar
- Ferwerda JA, Pattanaik SN, Shirley P, Greenberg DP: Model of visual adaptation for realistic image synthesis. Proceedings of the Computer Graphics Conference (SIGGRAPH '96), August 1996 249-258.Google Scholar
- Larson GW, Rushmeier H, Piatko C: A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Transactions on Visualization and Computer Graphics 1997,3(4):291-306. 10.1109/2945.646233View ArticleGoogle Scholar
- Schlick C: An adaptive sampling technique for multidimensional ray tracing. In Photorealistic Rendering in Computer Graphics. Springer, Berlin, Germany; 1994:21-29.View ArticleGoogle Scholar
- Chiu K, Herf M, Shirley P, Swamy S, Wang C, Zimmerman K: Spatially nonuniform scaling functions for high contrast images. Proceedings of the Graphics Interface, May 1993 245-253.Google Scholar
- Rahman Z, Jobson DJ, Woodell GA: Multiscale Retinex for color rendition and dynamic range compression. Signal and Image Processing, 1996, Proceedings of the SPIE 2847: 183-191.Google Scholar
- Pattanaik SN, Tumblin J, Yee H, Greenberg DP: Time-dependent visual adaptation for fast realistic image display. Proceedings of the Computer Graphics Conference (SIGGRAPH '00), 2000 47-54.Google Scholar
- Durand F, Dorsey J: Fast bilateral filtering for the display of high-dynamic-range images. Proceedings of the Computer Graphics Conference (ACM SIGGRAPH '02), July 2002 257-266.Google Scholar
- Reinhard E, Stark M, Shirley P, Ferwerda J: Photographic tone reproduction for digital images. Proceedings of the Computer Graphics Conference (SIGGRAPH '02), July 2002 267-276.Google Scholar
- Wang L, Horiuchi T, Kotera H, Tominaga S: HDR image compression and evaluation based on local adaptation using a retinal model. Journal of the Society for Information Display 2007,15(9):731-739. 10.1889/1.2785206View ArticleGoogle Scholar
- Meylan L, Alleysson D, Süsstrunk S: Model of retinal local adaptation for the tone mapping of color filter array images. Journal of the Optical Society of America A 2007,24(9):2807-2816. 10.1364/JOSAA.24.002807View ArticleGoogle Scholar
- Jobson DJ, Rahman Z, Woodell GA: Retinex image processing: improved fidelity to direct visual observation. Proceedings of the 4th Color Imaging Conference Color Science, Science, and Applications (CIC '96), November 1996 124-126.Google Scholar
- Pattanaik SN, Ferwerda JA, Fairchild MD, Greenberg DP: Multiscale model of adaptation and spatial vision for realistic image display. Proceedings of the Annual Conference on Computer Graphics (ACM SIGGRAPH '98), July 1998 287-298.Google Scholar
- Tumblin J, Turk G: ICIS: a boundary hierarchy for detail-preserving contrast reduction. In Proceedings of the Annual Conference on Computer Graphics (ACM SIGGRAPH '99), 1999. ACM; 83-90.Google Scholar
- Wang L, Horiuchi T, Kotera H: High dynamic range image compression by fast integrated surround retinex model. Journal of Imaging Science and Technology 2007,51(1):34-43. 10.2352/J.ImagingSci.Technol.(2007)51:1(34)View ArticleGoogle Scholar
- Vuylsteke P, Schoeters E: Method and apparatus for contrast enhancement. 1998. US Patent no. 5,805,721Google Scholar
- Li Y, Sharan L, Adelson EH: Compressing and companding high dynamic range with subband architectures. Proceedings of the Computer Graphics Conference (SIGGRAPH '05), August 2005 836-844.Google Scholar
- Naka KI, Rushton WA: S-potentials from luminosity units in the retina of fish (Cyprinidae). Journal of Physiology 1966,185(3):587-599.View ArticleGoogle Scholar
- Dowling JE: The Retina: An Approachable Part of the Brain. Belknap Press, Cambridge, Mass, USA; 1987.Google Scholar
- Tomasi C, Manduchi R: Bilateral filtering for gray and color images. Proceedings of the IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998 839-846.Google Scholar
- Land EH: The retinex. American Scientist 1964, 52: 247-264.Google Scholar
- McCann JJ: Retinex at 40. Journal of Electronic Imaging 2004, 13: 6-145. 10.1117/1.1645250View ArticleGoogle Scholar
- Rahman Z: Properties of a Center/Surround Retinex—part 1. Signal Processing Design. NASA Contractor Report 1995., (198194):Google Scholar
- Fairchild MD: The HDR photographic survey. Proceedings of the 15th Color Imaging Conference: Color Science and Engineering Systems, Technologies, and Applications (CIC '07), October 2007 233-238.Google Scholar
- Tominaga S: Multichannel vision system for estimating surface and illumination functions. Journal of the Optical Society of America A 1996,13(11):2163-2173. 10.1364/JOSAA.13.002163View ArticleGoogle Scholar
- Horiuchi T, Fu YQ, Tominaga S: Perceptual and colorimetric evaluations of HDR rendering with/without real-world scenes. Proceedings of the Congress of the International Colour Association (AIC 09), 2009, Sydney, AustraliaGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.