Skip to main content

Advertisement

EURASIP Journal on Image and Video Processing
EURASIP Journal on Image and Video Processing Cover Image

Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing

EURASIP Journal on Image and Video Processing volume 2007, Article number: 018019 (2007) Cite this article

Article metrics

  • 1146 Accesses

  • 5 Citations

Abstract

Vision is one of the main sources through which people obtain information from the world, but unfortunately, visually-impaired people are partially or completely deprived of this type of information. With the help of computer technologies, people with visual impairment can independently access digital textual information by using text-to-speech and text-to-Braille software. However, in general, there still exists a major barrier for people who are blind to access the graphical information independently in real-time without the help of sighted people. In this paper, we propose a novel multi-level and multi-modal approach aiming at addressing this challenging and practical problem, with the key idea being semantic-aware visual-to-tactile conversion through semantic image categorization and segmentation, and semantic-driven image simplification. An end-to-end prototype system was built based on the approach. We present the details of the approach and the system, report sample experimental results with realistic data, and compare our approach with current typical practice.

[12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152]

References

  1. 1.

    JAWS: http://www.freedomscientific.com/fs_products/JAWS_HQ.asp

  2. 2.

    Tactile Graphics Project at University of Washington: http://tactilegraphics.cs.washington.edu

  3. 3.

    Ladner RE, Ivory MY, Rao R, et al.: Automating tactile graphics translation. Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '05), October 2005, Baltimore, Md, USA 150-157.

  4. 4.

    Edman PK: Tactile Graphics. AFB Press, Sewickley, Pa, USA; 1992.

  5. 5.

    Wall RS, Corn AL: Production of textbooks and instructional materials in the United States. Journal of Visual Impairment & Blindness 2002,96(4):212-222.

  6. 6.

    Burger D: Improved access to computers for the visually handicapped: new prospects and principles. IEEE Transactions on Rehabilitation Engineering 1994,2(3):111-118. 10.1109/86.331560

  7. 7.

    Corn AL, Wall RS: Training and availability of Braille transcribers in the United States. Journal of Visual Impairment & Blindness 2002,96(4):223-232.

  8. 8.

    American Foundation for the Blind : Survey on the production of textbooks and instructional materials. 2000.

  9. 9.

    Hinton R: First introduction to tactiles. British Journal of Visual Impairment 1991,9(3):79-82. 10.1177/026461969100900304

  10. 10.

    Akamatsu M, MacKenzie IS, Hasbroucq T: A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics 1995,38(4):816-827. 10.1080/00140139508925152

  11. 11.

    Cavanagh P, Kennedy JM, Pelli DG, Palomares M: Close encounters: details veto depth from shadows. Science 2000, 287: 2421.

  12. 12.

    D'Angiulli A, Kennedy JM: Children's tactual exploration and copying without vision. International Journal of Rehabilitation Research 2001,24(3):233-234. 10.1097/00004356-200109000-00009

  13. 13.

    Hatwell Y, Marinez-Sarrochi F: The tactile reading of maps and drawings, and the access of blind people to works of art. In Touching for Knowing. Edited by: Hatwell Y, Streri A, Gentaz E. John Benjamings, Amsterdam, The Netherlands; 2003:255-273.

  14. 14.

    Heller MA, Kennedy JM: Perspective taking, pictures, and the blind. Perception & Psychophysics 1990,48(5):459-466. 10.3758/BF03211590

  15. 15.

    Kennedy JM, Juricevic I: Haptics and projection: drawings by Tracy, a blind adult. Perception 2003,32(9):1059-1071. 10.1068/p3425

  16. 16.

    Kennedy JM, Bai J: Haptic pictures: fit judgments predict identification, recognition memory, and confidence. Perception 2002,31(8):1013-1026. 10.1068/p3259

  17. 17.

    Kennedy JM, Merkas C: Depictions of motion devised by a blind person. Psychonomic Bulletin & Review 2000,7(4):700-706. 10.3758/BF03213009

  18. 18.

    Kennedy JM: Drawings by the blind: sighted children and adults judge their sequence of development. Visual Arts Research 1984, 10: 1-6.

  19. 19.

    Kokjer KJ: Information capacity of the human fingertip. IEEE Transactions on Systems, Man and Cybernetics 1987,17(1):100-102.

  20. 20.

    Magee LE, Kennedy JM: Exploring pictures tactually. Nature 1980, 283: 287-288. 10.1038/283287a0

  21. 21.

    Merabet L, Rizzo J, Amedi A, Somers DC, Pascual-Leone A: Opinion: what blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience 2005,6(1):71-77. 10.1038/nrn1586

  22. 22.

    Pathak K, Pring L: Tactual picture recognition in congenitally blind and sighted children. Applied Cognitive Psychology 1989, 3: 337-350. 10.1002/acp.2350030405

  23. 23.

    Bliss J, Katcher M, Rogers C, Shepard R: Optical-to-tactile image conversion for the blind. IEEE Transactions on Man Machine Systems 1970,11(1):58-65.

  24. 24.

    Stein D: The Optacon: Past, Present, and Future. National Federation of the Blind (NFB): http://www.nfb.org/Images/nfb/Publications/bm/bm98/bm980506.htm

  25. 25.

    Collins CC, Bach-y-Rita P: Transmission of pictorial information through the skin. Advances in Biological and Medical Physics 1973, 14: 285-315.

  26. 26.

    Tiger Embosser: http://www.enablemart.com

  27. 27.

    Way T, Barner K: Automatic visual to tactile translation—part I: human factors, access methods, and image manipulation. IEEE Transactions on Rehabilitation Engineering 1997,5(1):81-94. 10.1109/86.559353

  28. 28.

    Way T, Barner K: Automatic visual to tactile translation—part II: evaluation of the TACTile image creation system. IEEE Transactions on Rehabilitation Engineering 1997,5(1):95-105. 10.1109/86.559354

  29. 29.

    Ivory MY, Martin AP, Megraw R, Slabosky B: Augmented cognition: an approach to increasing universal benefit from information technology. Proceedings of the 1st International Conference on Augmented Cognition, July 2005, Las Vegas, Nev, USA

  30. 30.

    Ando H, Miki T, Inami M, Maeda T: The nail-mounted tactile display for the behavior modeling. Proceedings of ACM SIGGRAPH Conference Abstracts and Applications, July 2002, San Antonio, Tex, USA 264.

  31. 31.

    Nojima T, Sekiguchi D, Inami M, Tachi S: The SmartTool: a system for augmented reality of haptics. Proceedings of IEEE Virtual Reality Conference (VR '02), March 2002, Orlando, Fla, USA 67-72.

  32. 32.

    SmartTouch: http://www.star.t.u-tokyo.ac.jp/projects/smarttouch

  33. 33.

    http://kaz.med.wisc.edu/TDU.htm

  34. 34.

    http://www.eurekalert.org/pub_releases/2004-06/uom-aeo060204.php

  35. 35.

    Heyes AD: Human navigation by sound. Physics in Technology 1983,14(2):68-75. 10.1088/0305-4624/14/2/I02

  36. 36.

    Meijer PBL: An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering 1992,39(2):112-121. 10.1109/10.121642

  37. 37.

    iCARE Haptics: http://cubic.asu.edu/icare/reader.html

  38. 38.

    Lenay C, Canu S, Villon P: Technology and perception: the contribution of sensory substitution systems. Proceedings of the 2nd International Conference on Cognitive Technology, August 1997, Aizu, Japan 44-53.

  39. 39.

    Ammar AA, Gapenne O, Lenay C, Stewart JJ: Effect of bimodality on the perception of 2D forms by means of a specific assistive technology for blind persons. Proceedings of the Conference on Assistive Technology for Vision and Hearing Impairment (CVHI '2002), August 2002, Grenade, Espagne 45-52.

  40. 40.

    http://dots.physics.orst.edu

  41. 41.

    Xu X, Li B: Multiple-class multiple-instance learning for automated image categorization. to appear in Internal Journal of Image and Graphics

  42. 42.

    Li J, Wang JZ: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003,25(9):1075-1088. 10.1109/TPAMI.2003.1227984

  43. 43.

    Wang JZ, Li J, Wiederhold G: SIMPLIcity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(9):947-963. 10.1109/34.955109

  44. 44.

    Maron O: Learning from ambiguity, Doctoral dissertation. Massachusetts Institute of Technology, AI Technical Report 1639, Cambridge, Mass, USA; 1998.

  45. 45.

    Maron O, Lozano-Pérez T: A framework for multiple-instance learning. In Advances in Neural Information Processings Systems. Volume 10. MIT Press, Cambridge, Mass, USA; 1998.

  46. 46.

    Zhang Q, Goldman SA: EM-DD: an improved multiple-instance learning technique. In Advances in Neural Information Processing Systems. Volume 14. MIT Press, Cambridge, Mass, USA; 2002:1073-1080.

  47. 47.

    Amar RA, Dooly DR, Goldman SA, Zhang Q: Multiple-instance learning of real-valued data. In Proceedings of the 18th International Conference on Machine Learning (ICML '01), November 2001, San Francisco, Calif, USA. Morgan Kaufmann; 3-10.

  48. 48.

    Chen YX, Wang JZ: Image categorization by learning and reasoning with regions. Journal of Machine Learning Research 2004, 5: 913-939.

  49. 49.

    LibSVM: http://www.csie.ntu.edu.tw/~cjlin/libsvm

  50. 50.

    Wang Z, Li C: Building detection and recognition via the improved HOUGH transform. Proceedings of the International Computer Congress on Wavelet Analysis and Its Applications, and Active Media Technology, 2004 2: 1075-1080.

  51. 51.

    Krishnan NC, Li B, Panchanathan S: Detecting and classifying frontal, back, and profile views of humans. Proceedings of the International Conference on Computer Vision Theory and Application (VISAPP '07), March 2007, Barcelona, Spain

  52. 52.

    NIST: http://www.itl.nist.gov/div895/isis/braille.html

Download references

Author information

Affiliations

  1. Department of Computer Science and Engineering, School of Computing and Informatics, Arizona State University, Tempe, AZ, 85287-8809, USA

    • Zheshen Wang
    • , Xinyu Xu
    •  & Baoxin Li

Authors

  1. Search for Zheshen Wang in:

  2. Search for Xinyu Xu in:

  3. Search for Baoxin Li in:

Corresponding author

Correspondence to Zheshen Wang.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Wang, Z., Xu, X. & Li, B. Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing. J Image Video Proc 2007, 018019 (2007) doi:10.1155/2007/18019

Download citation

Keywords

  • Computer Vision
  • Visual Impairment
  • Computer Technology
  • Image Categorization
  • Textual Information