Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing
EURASIP Journal on Image and Video Processing volume 2007, Article number: 018019 (2007)
Vision is one of the main sources through which people obtain information from the world, but unfortunately, visually-impaired people are partially or completely deprived of this type of information. With the help of computer technologies, people with visual impairment can independently access digital textual information by using text-to-speech and text-to-Braille software. However, in general, there still exists a major barrier for people who are blind to access the graphical information independently in real-time without the help of sighted people. In this paper, we propose a novel multi-level and multi-modal approach aiming at addressing this challenging and practical problem, with the key idea being semantic-aware visual-to-tactile conversion through semantic image categorization and segmentation, and semantic-driven image simplification. An end-to-end prototype system was built based on the approach. We present the details of the approach and the system, report sample experimental results with realistic data, and compare our approach with current typical practice.
Tactile Graphics Project at University of Washington: http://tactilegraphics.cs.washington.edu
Ladner RE, Ivory MY, Rao R, et al.: Automating tactile graphics translation. Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '05), October 2005, Baltimore, Md, USA 150-157.
Edman PK: Tactile Graphics. AFB Press, Sewickley, Pa, USA; 1992.
Wall RS, Corn AL: Production of textbooks and instructional materials in the United States. Journal of Visual Impairment & Blindness 2002,96(4):212-222.
Burger D: Improved access to computers for the visually handicapped: new prospects and principles. IEEE Transactions on Rehabilitation Engineering 1994,2(3):111-118. 10.1109/86.331560
Corn AL, Wall RS: Training and availability of Braille transcribers in the United States. Journal of Visual Impairment & Blindness 2002,96(4):223-232.
American Foundation for the Blind : Survey on the production of textbooks and instructional materials. 2000.
Hinton R: First introduction to tactiles. British Journal of Visual Impairment 1991,9(3):79-82. 10.1177/026461969100900304
Akamatsu M, MacKenzie IS, Hasbroucq T: A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics 1995,38(4):816-827. 10.1080/00140139508925152
Cavanagh P, Kennedy JM, Pelli DG, Palomares M: Close encounters: details veto depth from shadows. Science 2000, 287: 2421.
D'Angiulli A, Kennedy JM: Children's tactual exploration and copying without vision. International Journal of Rehabilitation Research 2001,24(3):233-234. 10.1097/00004356-200109000-00009
Hatwell Y, Marinez-Sarrochi F: The tactile reading of maps and drawings, and the access of blind people to works of art. In Touching for Knowing. Edited by: Hatwell Y, Streri A, Gentaz E. John Benjamings, Amsterdam, The Netherlands; 2003:255-273.
Heller MA, Kennedy JM: Perspective taking, pictures, and the blind. Perception & Psychophysics 1990,48(5):459-466. 10.3758/BF03211590
Kennedy JM, Juricevic I: Haptics and projection: drawings by Tracy, a blind adult. Perception 2003,32(9):1059-1071. 10.1068/p3425
Kennedy JM, Bai J: Haptic pictures: fit judgments predict identification, recognition memory, and confidence. Perception 2002,31(8):1013-1026. 10.1068/p3259
Kennedy JM, Merkas C: Depictions of motion devised by a blind person. Psychonomic Bulletin & Review 2000,7(4):700-706. 10.3758/BF03213009
Kennedy JM: Drawings by the blind: sighted children and adults judge their sequence of development. Visual Arts Research 1984, 10: 1-6.
Kokjer KJ: Information capacity of the human fingertip. IEEE Transactions on Systems, Man and Cybernetics 1987,17(1):100-102.
Magee LE, Kennedy JM: Exploring pictures tactually. Nature 1980, 283: 287-288. 10.1038/283287a0
Merabet L, Rizzo J, Amedi A, Somers DC, Pascual-Leone A: Opinion: what blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience 2005,6(1):71-77. 10.1038/nrn1586
Pathak K, Pring L: Tactual picture recognition in congenitally blind and sighted children. Applied Cognitive Psychology 1989, 3: 337-350. 10.1002/acp.2350030405
Bliss J, Katcher M, Rogers C, Shepard R: Optical-to-tactile image conversion for the blind. IEEE Transactions on Man Machine Systems 1970,11(1):58-65.
Stein D: The Optacon: Past, Present, and Future. National Federation of the Blind (NFB): http://www.nfb.org/Images/nfb/Publications/bm/bm98/bm980506.htm
Collins CC, Bach-y-Rita P: Transmission of pictorial information through the skin. Advances in Biological and Medical Physics 1973, 14: 285-315.
Tiger Embosser: http://www.enablemart.com
Way T, Barner K: Automatic visual to tactile translation—part I: human factors, access methods, and image manipulation. IEEE Transactions on Rehabilitation Engineering 1997,5(1):81-94. 10.1109/86.559353
Way T, Barner K: Automatic visual to tactile translation—part II: evaluation of the TACTile image creation system. IEEE Transactions on Rehabilitation Engineering 1997,5(1):95-105. 10.1109/86.559354
Ivory MY, Martin AP, Megraw R, Slabosky B: Augmented cognition: an approach to increasing universal benefit from information technology. Proceedings of the 1st International Conference on Augmented Cognition, July 2005, Las Vegas, Nev, USA
Ando H, Miki T, Inami M, Maeda T: The nail-mounted tactile display for the behavior modeling. Proceedings of ACM SIGGRAPH Conference Abstracts and Applications, July 2002, San Antonio, Tex, USA 264.
Nojima T, Sekiguchi D, Inami M, Tachi S: The SmartTool: a system for augmented reality of haptics. Proceedings of IEEE Virtual Reality Conference (VR '02), March 2002, Orlando, Fla, USA 67-72.
Heyes AD: Human navigation by sound. Physics in Technology 1983,14(2):68-75. 10.1088/0305-4624/14/2/I02
Meijer PBL: An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering 1992,39(2):112-121. 10.1109/10.121642
iCARE Haptics: http://cubic.asu.edu/icare/reader.html
Lenay C, Canu S, Villon P: Technology and perception: the contribution of sensory substitution systems. Proceedings of the 2nd International Conference on Cognitive Technology, August 1997, Aizu, Japan 44-53.
Ammar AA, Gapenne O, Lenay C, Stewart JJ: Effect of bimodality on the perception of 2D forms by means of a specific assistive technology for blind persons. Proceedings of the Conference on Assistive Technology for Vision and Hearing Impairment (CVHI '2002), August 2002, Grenade, Espagne 45-52.
Xu X, Li B: Multiple-class multiple-instance learning for automated image categorization. to appear in Internal Journal of Image and Graphics
Li J, Wang JZ: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003,25(9):1075-1088. 10.1109/TPAMI.2003.1227984
Wang JZ, Li J, Wiederhold G: SIMPLIcity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(9):947-963. 10.1109/34.955109
Maron O: Learning from ambiguity, Doctoral dissertation. Massachusetts Institute of Technology, AI Technical Report 1639, Cambridge, Mass, USA; 1998.
Maron O, Lozano-Pérez T: A framework for multiple-instance learning. In Advances in Neural Information Processings Systems. Volume 10. MIT Press, Cambridge, Mass, USA; 1998.
Zhang Q, Goldman SA: EM-DD: an improved multiple-instance learning technique. In Advances in Neural Information Processing Systems. Volume 14. MIT Press, Cambridge, Mass, USA; 2002:1073-1080.
Amar RA, Dooly DR, Goldman SA, Zhang Q: Multiple-instance learning of real-valued data. In Proceedings of the 18th International Conference on Machine Learning (ICML '01), November 2001, San Francisco, Calif, USA. Morgan Kaufmann; 3-10.
Chen YX, Wang JZ: Image categorization by learning and reasoning with regions. Journal of Machine Learning Research 2004, 5: 913-939.
Wang Z, Li C: Building detection and recognition via the improved HOUGH transform. Proceedings of the International Computer Congress on Wavelet Analysis and Its Applications, and Active Media Technology, 2004 2: 1075-1080.
Krishnan NC, Li B, Panchanathan S: Detecting and classifying frontal, back, and profile views of humans. Proceedings of the International Conference on Computer Vision Theory and Application (VISAPP '07), March 2007, Barcelona, Spain
About this article
Cite this article
Wang, Z., Xu, X. & Li, B. Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing. J Image Video Proc 2007, 018019 (2007). https://doi.org/10.1155/2007/18019
- Computer Vision
- Visual Impairment
- Computer Technology
- Image Categorization
- Textual Information