HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments

HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.

Publications

teaser image of Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

ACM Transactions on Accessible Computing (TACCESS), 2016.
Keywords: accessibility, real-time OCR, visual impairments, wearables

teaser image of The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.
Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind

teaser image of Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS), 2015.
Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch


Videos

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind


Talks

Cited By

  • Designing to Support and Extend the Competencies of People With Visual Impairments. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Gisela Reyes Cruz. [doi]
  • Three-Dimensional Nonvisual Directional Guidance for People With Visual Impairments. 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). Seung A Chung, Kyungyeon Lee, Sohyeon Park, and Uran Oh. [doi]
  • FingerReader. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Nanayakkara. [doi]
  • Social Sensemaking With AI: Designing an Open-Ended AI Experience With a Blind Child. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Cecily Morrison, Edward Cutrell, Martin Grayson, Anja Thieme, Alex Taylor, Geert Roumen, Camilla Longden, Sebastian Tschiatschek, Rita Faia Marques, and Abigail Sellen. [doi]
  • HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments. University of Maryland, College Park.Lee Stearns. [doi]
  • Digital Digits: Designing Assistive Finger Augmentation Devices. Massachusetts Institute of Technology. Roy Shilkrot. [doi]
  • AiSee. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Roger Boldu, Denys J.C. Matthies, Haimo Zhang, and Suranga Nanayakkara. [doi]
  • HearIt: Auditory-Cue-Based Audio Playback Control to Facilitate Information Browsing in Lecture Audio. Applied Sciences. Jeongmin Hong, Hyesoo Jeon, Hana Lee, Daehyun Kim, and Minsam Ko. [doi]
  • Printed Texts Tracking and Following for a Finger-Wearable Electro-Braille System Through Opto-Electrotactile Feedback. https://arxiv.org/abs/2109.02385.pdf. Mehdi Rahimi, Yantao Shen, Zhiming Liu, and Fang Jiang. [doi]
  • Stay In Touch