A Framework for the Assessment of Temporal Artifacts in Medium Frame-Rate Binary Video Halftones
© Hamood-Ur Rehman and Brian L. Evans. 2010
Received: 1 May 2010
Accepted: 2 August 2010
Published: 17 August 2010
Display of a video having a higher number of bits per pixel than that available on the display device requires quantization prior to display. Video halftoning performs this quantization so as to reduce visibility of certain artifacts. In many cases, visibility of one set of artifacts is decreased at the expense of increasing the visibility of another set. In this paper, we focus on two key temporal artifacts, flicker and dirty-window-effect, in binary video halftones. We quantify the visibility of these two artifacts when the video halftone is displayed at medium frame rates (15 to 30 frames per second). We propose new video halftoning methods to reduce visibility of these artifacts. The proposed contributions are (1) an enhanced measure of perceived flicker, (2) a new measure of perceived dirty-window-effect, (3) a new video halftoning method to reduce flicker, and (4) a new video halftoning method to reduce dirty-window-effect.
Bit-depth reduction must be performed when the number of bits/pixel (bit-depth) of the original video data is higher than the bit-depth available on the display device. Halftoning is a process that can perform this quantization. The original, full bit-depth video is called the continuous-tone video, and the reduced bit-depth video is called the halftone video. Bit-depth reduction results in quantization artifacts.
Binary halftone videos can suffer from both spatial and temporal artifacts. In the case of binary halftone videos produced from grayscale continuous-tone videos, there are two key temporal artifacts. These temporal artifacts are flicker and dirty-window-effect (DWE). Of these two temporal artifacts, halftone flicker has received more attention in publications on video halftoning [1–5]. Hilgenberg et al. briefly discuss the DWE artifact in . They have, however, not used the term dirty-window-effect to refer to this particular artifact.
The DWE refers to the temporal artifact that gives a human viewer the perception of viewing objects, in the halftone video, through a "dirty" transparent medium, such as a window. The artifact is usually disturbing to the viewer because it gives the perception as if a pattern were laid on top of the actual video. Like other artifacts, dirty-window-effect contributes to a degraded viewing experience of the viewer. Although this artifact is known and has been referred to in the published literature , as far as we know, a quantitative perceptual criteria to assess this artifact has not been published. The artifact has been evaluated qualitatively in .
In contrast to DWE, which is observed due to binary pixels not toggling in enough numbers in response to a changing scene, flicker is typically observed due to too many binary pixels toggling their values in spatial areas that do not exhibit "significant" perceptual change between successive (continuous-tone) frames. Depending on the type of display, flicker can appear as full-field flicker or as scintillations. As a temporal artifact, halftone flicker can appear unpleasant to a viewer. On some devices, it can also result in higher power consumption . Moreover, if the halftone video is to be compressed for storage or transmission, higher flicker can reduce the compression efficiency [2, 3]. Evaluation of flicker has been discussed in [2–5]. Flicker has been referred to as high frequency temporal noise in . A recent approach to form a perceptual estimate of flicker has been discussed in .
For reasons discussed above, it is desirable to reduce these temporal artifacts in the halftone videos. Therefore, perceptual quantitative measures for evaluating these artifacts are desirable. Quantitative assessment of temporal artifacts can facilitate comparison of binary halftone videos produced using different algorithms. Temporal artifact quality assessment criteria can also be combined with the assessment of spatial artifacts to form an overall quality assessment criteria for binary halftone videos. Video halftoning algorithm design can benefit from the temporal artifact evaluation criteria presented in this paper. The perception of temporal artifacts is dependent on the frame-rate at which the halftone video is viewed. For example, for medium frame rate (15 to 30 frames per second) binary halftone videos, flicker between successive halftone frames will correspond to temporal frequencies at which the human visual system (HVS) is sensitive .
In this paper, we present a framework for the quantitative evaluation of the temporal artifacts in medium frame rate binary halftone videos produced from grayscale continuous-tone videos. We utilize the proposed quality assessment framework to design video halftoning algorithms. The proposed contributions of this paper include ( ) an enhanced measure of perceived flicker, ( ) a new measure of perceived dirty-window-effect, ( ) a new video halftoning method to reduce flicker, and ( ) a new video halftoning method to reduce dirty-window-effect.
The rest of the paper is organized as follows. Flicker and dirty-window-effect in binary halftone videos are discussed in detail in Section 2. Section 3 presents the proposed technique to assess temporal artifacts. Section 3 also presents halftoning algorithms that reduce temporal artifacts based on the proposed quality assessment techniques. The paper concludes with a summary of the proposed contributions in Section 4.
2. Flicker and Dirty-Window-Effect
As discussed in the previous section, dirty-window-effect refers to the temporal artifact that causes the illusion of viewing the moving objects, in the halftone video, through a dirty window. In medium frame-rate binary halftone videos, the perception of dirty-window-effect depends primarily on both the continuous-tone and the corresponding halftone videos. Consider two successive continuous-tone frames and their corresponding halftone frames. Assume that some objects that appear in the first continuous-tone frame change their spatial position in the second, successive, continuous-tone frame, but the corresponding halftone frames do not "sufficiently" change in their halftone patterns at the spatial locations where the continuous-tone frames changed. When each of the two halftone frames is viewed independently, it represents a good perceptual approximation of its corresponding continuous-tone frame. However, when the two halftone frames are viewed in a sequence, if the change in their binary patterns does not "sufficiently" reflect the corresponding change in the continuous-tone frames, the halftone video can suffer from perceivable dirty-window-effect. DWE should not be visible if the successive continuous-tone frames are identical.
Now, consider a scenario where the values of grayscale pixels within a (spatial) region of a continuous-tone frame are close to the values of the corresponding pixels in the next (successive) continuous-tone frame. If such is the case, one would expect the corresponding binary halftone frames to have similar pixels values as well. However, it is possible that although each of the corresponding binary halftone frame is perceptually similar to its continuous-tone version, when viewed in a sequence the two successive halftone frames toggle their pixel values within the same spatial region. This can result in the perception of flicker.
3. Proposed Technique
Flicker and dirty-window-effect in a binary halftone video represent local phenomena. That is, their perception depends on both the temporal and spatial characteristics of the halftone video. Thus, flicker or DWE may be more observable in certain frames and in certain spatial locations of those frames. In our observation, the perception of DWE is higher if the moving objects (or regions) are relatively flat. This means that moving objects with higher spatial frequencies (or with higher degree of contrast) are less likely to cause the perception of DWE. Similarly, the perception of flicker is higher if the similar corresponding spatial regions of two successive halftone frames have higher low spatial frequency (or low contrast) content. It is interesting to note that for still image halftones, it has been reported that the nature of dither is most important in the flat regions of the image . This phenomenon is due to the spatial masking effects that hide the presence of noise in regions of the image that have high spatial frequencies or are textured.
Flicker and DWE are related. Reducing one artifact could result in an increase of the other. If halftone pixels toggle values between halftone frames within a spatial area that does not change much between continuous-tone frames, flicker might be observed at medium frame rates. If they do not toggle in spatial areas that change between successive frames or exhibit motion, DWE might be observed. To minimize both artifacts, a halftoning algorithm should produce halftone frames that have their pixels toggle values only in spatial regions that have a perceptual change (due to motion, e.g.) between the corresponding successive continuous-tone frames. Certain halftoning algorithms produce videos that have high DWE but low flicker. An example is a binary halftone video produced by using ordered-dither technique on each grayscale continuous-tone frame independently. Similarly, there are halftoning algorithms that produce videos with high flicker but low DWE. An example is a binary halftone video produced by halftoning each grayscale continuous-tone frame independently using Floyd and Steinberg  error diffusion algorithm.
3.1. Halftone Dirty-Window-Effect Evaluation
Dirty-window-effect performance of individual halftone frames can be represented as a plot of against frame number. The DWE performance of the entire halftone video is given by the single number DWE, the Perceptual DWE Index. The framework introduced thus far is quite general. We have not described the form of the function in (1). We have also not described how to calculate the arguments of this function. We provide these details next.
3.2. Experimental Results on DWE Assessment
3.3. Validation of the DWE Assessment Framework
3.4. Halftone Flicker Evaluation
Perceived Average Flicker can be plotted (against frame number) to evaluate flicker performance of individual halftone frames. Perceptual Flicker Index gives a single number representing flicker performance of the entire halftone video. Next, we present a particular instantiation of the framework discussed thus far.
Note that . This instantiation of the flicker assessment framework is depicted in Figure 9. In Figure 9, K, Q, and R each have a value of 1. L, and S have each a value of 0. P has a value of 1. The "Artifact Map" is . has the form described in . We do evaluate differently in this paper. For clarity, we repeat the description of as provided in . is a product of three terms. At pixel location , the first term measures the local similarity between the successive continuous-tone frames. A higher value of the first term, , will mean that the successive frames have a higher structural similarity in a local neighborhood of pixels centered at pixel location . This will in turn assign a higher weight to any flicker observed. This is desired because if the "local" scene does not change, perception of any flicker would be higher. The second term, , depends on the number of pixels that toggled in a neighborhood around (and including) pixel location . It gives us a measure of perceived flicker due to HVS filtering. Since the HVS is modeled as a low pass filter in this experiment, will have a relatively higher value, if the pixel toggles form a cluster as opposed to being dispersed. The third term, , measures the low contrast content in a local neighborhood centered at . A higher value of this term will result in higher value of perceived flicker. Finally, we incorporate the effect of scene changes by setting to a low value (zero, e.g.), if a scene change is detected between continuous-tone frames and . This is to account for temporal masking effects. For the results reported in this paper, we (manually) determined scene changes in the videos through visual inspection and manually set to zero whenever a scene change is determined to have occurred between successive continuous-tone frames and .
3.5. Experimental Results on Flicker Assessment
3.6. Validation of the Flicker Assessment Framework
As seen in (12), the amount of threshold perturbation is determined by , where is a constant that controls the effect of on . The threshold modulation is designed to reduce flicker in the halftone video.
In this paper, we presented a generalized framework for the perceptual assessment of two temporal artifacts in medium frame rate binary video halftones produced from grayscale continuous-tone videos. The two temporal artifacts discussed in this paper were referred to as halftone flicker and halftone dirty-window-effect. For the perceptual evaluation of each artifact, a particular instantiation of the generalized framework, was presented and the associated results were discussed. We also presented two new video halftoning algorithms which were designed by modifying existing video halftoning algorithms. The modifications were based on the perceptual quality assessment framework and were thus geared towards reducing the temporal artifacts. Results of comparisons between the halftone videos generated using the original and the modified algorithms were presented and discussed.
- Rehman H, Evans BL: Flicker assessment of low-to-medium frame-rate binary video halftones. Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI '10), May 2010, Austin, Tex, USA 185-188.Google Scholar
- Gotsman C: Halftoning of image sequences. Visual Computer 1993,9(5):255-266. 10.1007/BF01908448View ArticleGoogle Scholar
- Hsu C-Y, Lu C-S, Pei S-C: Video halftoning preserving temporal consistency. Proceedings of IEEE International Conference on Multimedia and Expo (ICME '07), July 2007 1938-1941.Google Scholar
- Hild H, Pins M: A 3-d error diffusion dither algorithm for half-tone animation on bitmap screens. In State-of-the-Art in Computer Animation. Springer, Berlin, Germany; 1989:181-190.View ArticleGoogle Scholar
- Sun Z: Video halftoning. IEEE Transactions on Image Processing 2006,15(3):678-686.View ArticleGoogle Scholar
- Hilgenberg DP, Flohr TJ, Atkins CB, Allebach JP, Bouman CA: Least-squares model-based video halftoning. Human Vision, Visual Processing, and Digital Display V, February 1994, San Jose, Calif, USA, Proceedings of SPIE 2179: 207-217.View ArticleGoogle Scholar
- Hsu C-Y, Lu C-S, Pei S-C: Power-scalable multi-layer halftone video display for electronic paper. Proceedings of IEEE International Conference on Multimedia and Expo (ICME '08), June 2008, Hannover, Germany 1445-1448.Google Scholar
- Robson J: Spatial and temporal contrast-sensitivity functions of the visual system. Journal of the Optical Society of America 1966,56(8):1141-1142. 10.1364/JOSA.56.001141View ArticleGoogle Scholar
- Ulichney RA: Void-and-cluster method for dither array generation. In Human Vision, Visual Processing, and Digital Display IV, February 1993, San Jose, Calif, USA, Proceedings of SPIE Edited by: Allebach JP, Rogowitz BE. 1913: 332-343.View ArticleGoogle Scholar
- R. P. I. Center for Image Processing Research, http://www.cipr.rpi.edu/resource/sequences/index.html
- Tam WJ, Stelmach LB, Wang L, Lauzon D, Gray P: Visual masking at video scene cuts. Human Vision, Visual Processing, and Digital Display VI, February 1995, Proceedings of SPIE 2411: 111-119.View ArticleGoogle Scholar
- Ulichney RA: Review of halftoning techniques. Color Imaging: Device-Independent Color, Color Hardcopy, and Graphic Arts V, January 2000, San Jose, Calif, USA, Proceedings of SPIE 3963: 378-391.Google Scholar
- Girod B: The information theoretical significance of spatial and temporal masking in video signals. Human Vision, Visual processing, and Digital Display, 1989, Proceedings of SPIE 1077: 178-187.View ArticleGoogle Scholar
- Floyd R, Steinberg L: An adaptive algorithm for spatial grayscale. Proceedings of SID International Symposium, Digest of Technical Papers, 1976 36-37.Google Scholar
- Wang Z, Bovik AC, Sheikh HR, Simoncelli EP: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 2004,13(4):600-612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar
- Pappas TN, Allebach JP, Neuhoff DL: Model-based digital halftoning. IEEE Signal Processing Magazine 2003,20(4):14-27. 10.1109/MSP.2003.1215228View ArticleGoogle Scholar
- Nasanen R: Visibility of halftone dot textures. IEEE Transactions on Systems, Man and Cybernetics 1984,14(6):920-924.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.