Skip to main content
EURASIP Journal on Image and Video Processing
Download PDF

Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing

EURASIP Journal on Image and Video Processing volume 2007, Article number: 018019 (2007) Cite this article

Abstract

Vision is one of the main sources through which people obtain information from the world, but unfortunately, visually-impaired people are partially or completely deprived of this type of information. With the help of computer technologies, people with visual impairment can independently access digital textual information by using text-to-speech and text-to-Braille software. However, in general, there still exists a major barrier for people who are blind to access the graphical information independently in real-time without the help of sighted people. In this paper, we propose a novel multi-level and multi-modal approach aiming at addressing this challenging and practical problem, with the key idea being semantic-aware visual-to-tactile conversion through semantic image categorization and segmentation, and semantic-driven image simplification. An end-to-end prototype system was built based on the approach. We present the details of the approach and the system, report sample experimental results with realistic data, and compare our approach with current typical practice.

[12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152]

References

  1. JAWS: http://www.freedomscientific.com/fs_products/JAWS_HQ.asp

  2. Tactile Graphics Project at University of Washington: http://tactilegraphics.cs.washington.edu

  3. Ladner RE, Ivory MY, Rao R, et al.: Automating tactile graphics translation. Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '05), October 2005, Baltimore, Md, USA 150-157.

    Chapter  Google Scholar 

  4. Edman PK: Tactile Graphics. AFB Press, Sewickley, Pa, USA; 1992.

    Google Scholar 

  5. Wall RS, Corn AL: Production of textbooks and instructional materials in the United States. Journal of Visual Impairment & Blindness 2002,96(4):212-222.

    Google Scholar 

  6. Burger D: Improved access to computers for the visually handicapped: new prospects and principles. IEEE Transactions on Rehabilitation Engineering 1994,2(3):111-118. 10.1109/86.331560

    Article  Google Scholar 

  7. Corn AL, Wall RS: Training and availability of Braille transcribers in the United States. Journal of Visual Impairment & Blindness 2002,96(4):223-232.

    Google Scholar 

  8. American Foundation for the Blind : Survey on the production of textbooks and instructional materials. 2000.

    Google Scholar 

  9. Hinton R: First introduction to tactiles. British Journal of Visual Impairment 1991,9(3):79-82. 10.1177/026461969100900304

    Article  Google Scholar 

  10. Akamatsu M, MacKenzie IS, Hasbroucq T: A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics 1995,38(4):816-827. 10.1080/00140139508925152

    Article  Google Scholar 

  11. Cavanagh P, Kennedy JM, Pelli DG, Palomares M: Close encounters: details veto depth from shadows. Science 2000, 287: 2421.

    Article  Google Scholar 

  12. D'Angiulli A, Kennedy JM: Children's tactual exploration and copying without vision. International Journal of Rehabilitation Research 2001,24(3):233-234. 10.1097/00004356-200109000-00009

    Article  Google Scholar 

  13. Hatwell Y, Marinez-Sarrochi F: The tactile reading of maps and drawings, and the access of blind people to works of art. In Touching for Knowing. Edited by: Hatwell Y, Streri A, Gentaz E. John Benjamings, Amsterdam, The Netherlands; 2003:255-273.

    Chapter  Google Scholar 

  14. Heller MA, Kennedy JM: Perspective taking, pictures, and the blind. Perception & Psychophysics 1990,48(5):459-466. 10.3758/BF03211590

    Article  Google Scholar 

  15. Kennedy JM, Juricevic I: Haptics and projection: drawings by Tracy, a blind adult. Perception 2003,32(9):1059-1071. 10.1068/p3425

    Article  Google Scholar 

  16. Kennedy JM, Bai J: Haptic pictures: fit judgments predict identification, recognition memory, and confidence. Perception 2002,31(8):1013-1026. 10.1068/p3259

    Article  Google Scholar 

  17. Kennedy JM, Merkas C: Depictions of motion devised by a blind person. Psychonomic Bulletin & Review 2000,7(4):700-706. 10.3758/BF03213009

    Article  Google Scholar 

  18. Kennedy JM: Drawings by the blind: sighted children and adults judge their sequence of development. Visual Arts Research 1984, 10: 1-6.

    Google Scholar 

  19. Kokjer KJ: Information capacity of the human fingertip. IEEE Transactions on Systems, Man and Cybernetics 1987,17(1):100-102.

    Article  Google Scholar 

  20. Magee LE, Kennedy JM: Exploring pictures tactually. Nature 1980, 283: 287-288. 10.1038/283287a0

    Article  Google Scholar 

  21. Merabet L, Rizzo J, Amedi A, Somers DC, Pascual-Leone A: Opinion: what blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience 2005,6(1):71-77. 10.1038/nrn1586

    Article  Google Scholar 

  22. Pathak K, Pring L: Tactual picture recognition in congenitally blind and sighted children. Applied Cognitive Psychology 1989, 3: 337-350. 10.1002/acp.2350030405

    Article  Google Scholar 

  23. Bliss J, Katcher M, Rogers C, Shepard R: Optical-to-tactile image conversion for the blind. IEEE Transactions on Man Machine Systems 1970,11(1):58-65.

    Article  Google Scholar 

  24. Stein D: The Optacon: Past, Present, and Future. National Federation of the Blind (NFB): http://www.nfb.org/Images/nfb/Publications/bm/bm98/bm980506.htm

  25. Collins CC, Bach-y-Rita P: Transmission of pictorial information through the skin. Advances in Biological and Medical Physics 1973, 14: 285-315.

    Article  Google Scholar 

  26. Tiger Embosser: http://www.enablemart.com

  27. Way T, Barner K: Automatic visual to tactile translation—part I: human factors, access methods, and image manipulation. IEEE Transactions on Rehabilitation Engineering 1997,5(1):81-94. 10.1109/86.559353

    Article  Google Scholar 

  28. Way T, Barner K: Automatic visual to tactile translation—part II: evaluation of the TACTile image creation system. IEEE Transactions on Rehabilitation Engineering 1997,5(1):95-105. 10.1109/86.559354

    Article  Google Scholar 

  29. Ivory MY, Martin AP, Megraw R, Slabosky B: Augmented cognition: an approach to increasing universal benefit from information technology. Proceedings of the 1st International Conference on Augmented Cognition, July 2005, Las Vegas, Nev, USA

    Google Scholar 

  30. Ando H, Miki T, Inami M, Maeda T: The nail-mounted tactile display for the behavior modeling. Proceedings of ACM SIGGRAPH Conference Abstracts and Applications, July 2002, San Antonio, Tex, USA 264.

    Google Scholar 

  31. Nojima T, Sekiguchi D, Inami M, Tachi S: The SmartTool: a system for augmented reality of haptics. Proceedings of IEEE Virtual Reality Conference (VR '02), March 2002, Orlando, Fla, USA 67-72.

    Google Scholar 

  32. SmartTouch: http://www.star.t.u-tokyo.ac.jp/projects/smarttouch

  33. http://kaz.med.wisc.edu/TDU.htm

  34. http://www.eurekalert.org/pub_releases/2004-06/uom-aeo060204.php

  35. Heyes AD: Human navigation by sound. Physics in Technology 1983,14(2):68-75. 10.1088/0305-4624/14/2/I02

    Article  Google Scholar 

  36. Meijer PBL: An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering 1992,39(2):112-121. 10.1109/10.121642

    Article  Google Scholar 

  37. iCARE Haptics: http://cubic.asu.edu/icare/reader.html

  38. Lenay C, Canu S, Villon P: Technology and perception: the contribution of sensory substitution systems. Proceedings of the 2nd International Conference on Cognitive Technology, August 1997, Aizu, Japan 44-53.

    Google Scholar 

  39. Ammar AA, Gapenne O, Lenay C, Stewart JJ: Effect of bimodality on the perception of 2D forms by means of a specific assistive technology for blind persons. Proceedings of the Conference on Assistive Technology for Vision and Hearing Impairment (CVHI '2002), August 2002, Grenade, Espagne 45-52.

    Google Scholar 

  40. http://dots.physics.orst.edu

  41. Xu X, Li B: Multiple-class multiple-instance learning for automated image categorization. to appear in Internal Journal of Image and Graphics

  42. Li J, Wang JZ: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003,25(9):1075-1088. 10.1109/TPAMI.2003.1227984

    Article  Google Scholar 

  43. Wang JZ, Li J, Wiederhold G: SIMPLIcity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(9):947-963. 10.1109/34.955109

    Article  Google Scholar 

  44. Maron O: Learning from ambiguity, Doctoral dissertation. Massachusetts Institute of Technology, AI Technical Report 1639, Cambridge, Mass, USA; 1998.

    Google Scholar 

  45. Maron O, Lozano-Pérez T: A framework for multiple-instance learning. In Advances in Neural Information Processings Systems. Volume 10. MIT Press, Cambridge, Mass, USA; 1998.

    Google Scholar 

  46. Zhang Q, Goldman SA: EM-DD: an improved multiple-instance learning technique. In Advances in Neural Information Processing Systems. Volume 14. MIT Press, Cambridge, Mass, USA; 2002:1073-1080.

    Google Scholar 

  47. Amar RA, Dooly DR, Goldman SA, Zhang Q: Multiple-instance learning of real-valued data. In Proceedings of the 18th International Conference on Machine Learning (ICML '01), November 2001, San Francisco, Calif, USA. Morgan Kaufmann; 3-10.

    Google Scholar 

  48. Chen YX, Wang JZ: Image categorization by learning and reasoning with regions. Journal of Machine Learning Research 2004, 5: 913-939.

    Google Scholar 

  49. LibSVM: http://www.csie.ntu.edu.tw/~cjlin/libsvm

  50. Wang Z, Li C: Building detection and recognition via the improved HOUGH transform. Proceedings of the International Computer Congress on Wavelet Analysis and Its Applications, and Active Media Technology, 2004 2: 1075-1080.

  51. Krishnan NC, Li B, Panchanathan S: Detecting and classifying frontal, back, and profile views of humans. Proceedings of the International Conference on Computer Vision Theory and Application (VISAPP '07), March 2007, Barcelona, Spain

    Google Scholar 

  52. NIST: http://www.itl.nist.gov/div895/isis/braille.html

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, School of Computing and Informatics, Arizona State University, Tempe, AZ, 85287-8809, USA

    Zheshen Wang, Xinyu Xu & Baoxin Li

Authors
  1. Zheshen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinyu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baoxin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheshen Wang.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Wang, Z., Xu, X. & Li, B. Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing. J Image Video Proc 2007, 018019 (2007). https://doi.org/10.1155/2007/18019

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1155/2007/18019

Keywords

  • Computer Vision
  • Visual Impairment
  • Computer Technology
  • Image Categorization
  • Textual Information