EURASIP Journal on Image and Video Processing

Impact Factor 1.742

EURASIP Journal on Image and Video Processing Cover Image
Research Article
Open Access

Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing

EURASIP Journal on Image and Video Processing20072007:018019

https://doi.org/10.1155/2007/18019

©  Zheshen Wang et al. 2007

Received: 15 January 2007

Accepted: 20 August 2007

Published: 15 November 2007

Abstract

Vision is one of the main sources through which people obtain information from the world, but unfortunately, visually-impaired people are partially or completely deprived of this type of information. With the help of computer technologies, people with visual impairment can independently access digital textual information by using text-to-speech and text-to-Braille software. However, in general, there still exists a major barrier for people who are blind to access the graphical information independently in real-time without the help of sighted people. In this paper, we propose a novel multi-level and multi-modal approach aiming at addressing this challenging and practical problem, with the key idea being semantic-aware visual-to-tactile conversion through semantic image categorization and segmentation, and semantic-driven image simplification. An end-to-end prototype system was built based on the approach. We present the details of the approach and the system, report sample experimental results with realistic data, and compare our approach with current typical practice.

[12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152]

Authors’ Affiliations

(1)
Department of Computer Science and Engineering, School of Computing and Informatics, Arizona State University

References

  1. JAWS: http://www.freedomscientific.com/fs_products/JAWS_HQ.asp
  2. Tactile Graphics Project at University of Washington: http://tactilegraphics.cs.washington.edu
  3. Ladner RE, Ivory MY, Rao R, et al.: Automating tactile graphics translation. Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '05), October 2005, Baltimore, Md, USA 150-157.View ArticleGoogle Scholar
  4. Edman PK: Tactile Graphics. AFB Press, Sewickley, Pa, USA; 1992.Google Scholar
  5. Wall RS, Corn AL: Production of textbooks and instructional materials in the United States. Journal of Visual Impairment & Blindness 2002,96(4):212-222.Google Scholar
  6. Burger D: Improved access to computers for the visually handicapped: new prospects and principles. IEEE Transactions on Rehabilitation Engineering 1994,2(3):111-118. 10.1109/86.331560View ArticleGoogle Scholar
  7. Corn AL, Wall RS: Training and availability of Braille transcribers in the United States. Journal of Visual Impairment & Blindness 2002,96(4):223-232.Google Scholar
  8. American Foundation for the Blind : Survey on the production of textbooks and instructional materials. 2000.Google Scholar
  9. Hinton R: First introduction to tactiles. British Journal of Visual Impairment 1991,9(3):79-82. 10.1177/026461969100900304View ArticleGoogle Scholar
  10. Akamatsu M, MacKenzie IS, Hasbroucq T: A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics 1995,38(4):816-827. 10.1080/00140139508925152View ArticleGoogle Scholar
  11. Cavanagh P, Kennedy JM, Pelli DG, Palomares M: Close encounters: details veto depth from shadows. Science 2000, 287: 2421.View ArticleGoogle Scholar
  12. D'Angiulli A, Kennedy JM: Children's tactual exploration and copying without vision. International Journal of Rehabilitation Research 2001,24(3):233-234. 10.1097/00004356-200109000-00009View ArticleGoogle Scholar
  13. Hatwell Y, Marinez-Sarrochi F: The tactile reading of maps and drawings, and the access of blind people to works of art. In Touching for Knowing. Edited by: Hatwell Y, Streri A, Gentaz E. John Benjamings, Amsterdam, The Netherlands; 2003:255-273.View ArticleGoogle Scholar
  14. Heller MA, Kennedy JM: Perspective taking, pictures, and the blind. Perception & Psychophysics 1990,48(5):459-466. 10.3758/BF03211590View ArticleGoogle Scholar
  15. Kennedy JM, Juricevic I: Haptics and projection: drawings by Tracy, a blind adult. Perception 2003,32(9):1059-1071. 10.1068/p3425View ArticleGoogle Scholar
  16. Kennedy JM, Bai J: Haptic pictures: fit judgments predict identification, recognition memory, and confidence. Perception 2002,31(8):1013-1026. 10.1068/p3259View ArticleGoogle Scholar
  17. Kennedy JM, Merkas C: Depictions of motion devised by a blind person. Psychonomic Bulletin & Review 2000,7(4):700-706. 10.3758/BF03213009View ArticleGoogle Scholar
  18. Kennedy JM: Drawings by the blind: sighted children and adults judge their sequence of development. Visual Arts Research 1984, 10: 1-6.Google Scholar
  19. Kokjer KJ: Information capacity of the human fingertip. IEEE Transactions on Systems, Man and Cybernetics 1987,17(1):100-102.View ArticleGoogle Scholar
  20. Magee LE, Kennedy JM: Exploring pictures tactually. Nature 1980, 283: 287-288. 10.1038/283287a0View ArticleGoogle Scholar
  21. Merabet L, Rizzo J, Amedi A, Somers DC, Pascual-Leone A: Opinion: what blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience 2005,6(1):71-77. 10.1038/nrn1586View ArticleGoogle Scholar
  22. Pathak K, Pring L: Tactual picture recognition in congenitally blind and sighted children. Applied Cognitive Psychology 1989, 3: 337-350. 10.1002/acp.2350030405View ArticleGoogle Scholar
  23. Bliss J, Katcher M, Rogers C, Shepard R: Optical-to-tactile image conversion for the blind. IEEE Transactions on Man Machine Systems 1970,11(1):58-65.View ArticleGoogle Scholar
  24. Stein D: The Optacon: Past, Present, and Future. National Federation of the Blind (NFB): http://www.nfb.org/Images/nfb/Publications/bm/bm98/bm980506.htm
  25. Collins CC, Bach-y-Rita P: Transmission of pictorial information through the skin. Advances in Biological and Medical Physics 1973, 14: 285-315.View ArticleGoogle Scholar
  26. Tiger Embosser: http://www.enablemart.com
  27. Way T, Barner K: Automatic visual to tactile translation—part I: human factors, access methods, and image manipulation. IEEE Transactions on Rehabilitation Engineering 1997,5(1):81-94. 10.1109/86.559353View ArticleGoogle Scholar
  28. Way T, Barner K: Automatic visual to tactile translation—part II: evaluation of the TACTile image creation system. IEEE Transactions on Rehabilitation Engineering 1997,5(1):95-105. 10.1109/86.559354View ArticleGoogle Scholar
  29. Ivory MY, Martin AP, Megraw R, Slabosky B: Augmented cognition: an approach to increasing universal benefit from information technology. Proceedings of the 1st International Conference on Augmented Cognition, July 2005, Las Vegas, Nev, USAGoogle Scholar
  30. Ando H, Miki T, Inami M, Maeda T: The nail-mounted tactile display for the behavior modeling. Proceedings of ACM SIGGRAPH Conference Abstracts and Applications, July 2002, San Antonio, Tex, USA 264.Google Scholar
  31. Nojima T, Sekiguchi D, Inami M, Tachi S: The SmartTool: a system for augmented reality of haptics. Proceedings of IEEE Virtual Reality Conference (VR '02), March 2002, Orlando, Fla, USA 67-72.Google Scholar
  32. SmartTouch: http://www.star.t.u-tokyo.ac.jp/projects/smarttouch
  33. http://kaz.med.wisc.edu/TDU.htm
  34. http://www.eurekalert.org/pub_releases/2004-06/uom-aeo060204.php
  35. Heyes AD: Human navigation by sound. Physics in Technology 1983,14(2):68-75. 10.1088/0305-4624/14/2/I02View ArticleGoogle Scholar
  36. Meijer PBL: An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering 1992,39(2):112-121. 10.1109/10.121642View ArticleGoogle Scholar
  37. iCARE Haptics: http://cubic.asu.edu/icare/reader.html
  38. Lenay C, Canu S, Villon P: Technology and perception: the contribution of sensory substitution systems. Proceedings of the 2nd International Conference on Cognitive Technology, August 1997, Aizu, Japan 44-53.Google Scholar
  39. Ammar AA, Gapenne O, Lenay C, Stewart JJ: Effect of bimodality on the perception of 2D forms by means of a specific assistive technology for blind persons. Proceedings of the Conference on Assistive Technology for Vision and Hearing Impairment (CVHI '2002), August 2002, Grenade, Espagne 45-52.Google Scholar
  40. http://dots.physics.orst.edu
  41. Xu X, Li B: Multiple-class multiple-instance learning for automated image categorization. to appear in Internal Journal of Image and GraphicsGoogle Scholar
  42. Li J, Wang JZ: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003,25(9):1075-1088. 10.1109/TPAMI.2003.1227984View ArticleGoogle Scholar
  43. Wang JZ, Li J, Wiederhold G: SIMPLIcity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(9):947-963. 10.1109/34.955109View ArticleGoogle Scholar
  44. Maron O: Learning from ambiguity, Doctoral dissertation. Massachusetts Institute of Technology, AI Technical Report 1639, Cambridge, Mass, USA; 1998.Google Scholar
  45. Maron O, Lozano-Pérez T: A framework for multiple-instance learning. In Advances in Neural Information Processings Systems. Volume 10. MIT Press, Cambridge, Mass, USA; 1998.Google Scholar
  46. Zhang Q, Goldman SA: EM-DD: an improved multiple-instance learning technique. In Advances in Neural Information Processing Systems. Volume 14. MIT Press, Cambridge, Mass, USA; 2002:1073-1080.Google Scholar
  47. Amar RA, Dooly DR, Goldman SA, Zhang Q: Multiple-instance learning of real-valued data. In Proceedings of the 18th International Conference on Machine Learning (ICML '01), November 2001, San Francisco, Calif, USA. Morgan Kaufmann; 3-10.Google Scholar
  48. Chen YX, Wang JZ: Image categorization by learning and reasoning with regions. Journal of Machine Learning Research 2004, 5: 913-939.Google Scholar
  49. LibSVM: http://www.csie.ntu.edu.tw/~cjlin/libsvm
  50. Wang Z, Li C: Building detection and recognition via the improved HOUGH transform. Proceedings of the International Computer Congress on Wavelet Analysis and Its Applications, and Active Media Technology, 2004 2: 1075-1080.Google Scholar
  51. Krishnan NC, Li B, Panchanathan S: Detecting and classifying frontal, back, and profile views of humans. Proceedings of the International Conference on Computer Vision Theory and Application (VISAPP '07), March 2007, Barcelona, SpainGoogle Scholar
  52. NIST: http://www.itl.nist.gov/div895/isis/braille.html

Copyright

© Zheshen Wang et al. 2007

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.