HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.
Publications ACM Transactions on Accessible Computing (TACCESS), 2016.
Keywords: accessibility, real-time OCR, visual impairments, wearables, XR interaction
Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.
Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind
Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.
Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch
Cited By Understanding the Two-Step Nonvisual Omnidirectional Guidance for Target Acquisition in 3D Spaces 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). SeungA Chung , Kyungyeon Lee , and Uran Oh . source | cite | search Compact Microscopy Systems With Non-conventional Optical Techniques 9. Tahseen Kamal . source | cite | search Improving the Accessibility of Mobile OCR Apps Via Interactive Modalities ACM Transactions on Accessible Computing. Michael Cutter and Roberto Manduchi . source | cite | search Imagining Artificial Intelligence Applications With People With Visual Disabilities Using Tactile Ideation Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Cecily Morrison , Edward Cutrell , Anupama Dhareshwar , Kevin Doherty , Anja Thieme , and Alex Taylor . source | cite | search FingerReader: A Wearable Device to Explore Printed Text on the Go Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Roy Shilkrot , Jochen Huber , Wong Meng Ee , Pattie Maes , and Suranga Nanayakkara . source | cite | search FingerReader: A Finger-Worn Assistive Augmentation Assistive Augmentation. Roy Shilkrot , Jochen Huber , Roger Boldu , Pattie Maes , and Suranga Nanayakkara . source | cite | search Elastic Documents: Coupling Text and Tables Through Contextual Visualizations for Enhanced Document Reading IEEE Transactions on Visualization and Computer Graphics. Sriram Karthik Badam , Zhicheng Liu , and Niklas Elmqvist . source | cite | search Towards Automatic Diagram Description for the Blind Proceedings of the 11th International Convention on Rehabilitation Engineering and Assistive Technology. Seehait Chockthanyawat , Ekapol Chuangsuwanich , Atiwong Suchato , and Proadpran Punyabukkana . source | cite | search Computer Vision Methods for Visually Impaired People Fifteenth International Conference on Correlation Optics. Rostyslav Zatserkovnyi , Volodymyr Maik , and Roksoliana Zatserkovna . source | cite | search Opto-electrotactile Feedback Enabled Text-line Tracking Control for a Finger-wearable Reading Aid for the Blind 2022 International Conference on Robotics and Automation (ICRA). Mehdi Rahimi , Yantao Shen , Cong Peng , Zhiming Liu , and Fang Jiang . source | cite | search Computational Glasses: Repurposing Augmented Reality Glasses for Vision Assistance University of Otago. Jonathan Sutton . source | cite | search Stimulation of Numerical Skills in Children With Visual Impairments Using Image Recognition Procedia Computer Science. Jos{\'{e}} Antonio Taqu{\'{\i}}a Gutiérreza . source | cite | search Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement GI '16: Proceedings of the 42nd Graphics Interface Conference. Jonggi Hong , Lee Stearns , Jon E. Froehlich , David Ross , and Leah Findlater . source | cite | search Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Jonggi Hong , Alisha Pradhan , Jon E. Froehlich , and Leah Findlater . source | cite | search Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera 5. Alexander J. Medeiros , Lee Stearns , Leah Findlater , Chuan Chen , and Jon E. Froehlich . source | cite | search Towards a Natural User Interface to Support People With Visual Impairments in Detecting Colors 2016 23rd International Conference on Pattern Recognition (ICPR). Sergio Mascetti , Chiara Rossetti , Andrea Gerino , Cristian Bernareggi , Lorenzo Picinali , and Alessandro Rizzi . source | cite | search HapAR: Handy Intelligent Multimodal Haptic and Audio-Based Mobile AR Navigation for the Visually Impaired Technological Trends in Improved Mobility of the Visually Impaired. EAI/Springer Innovations in Communication and Computing. Ahmad Hoirul Basori . source | cite | search Investigating Microinteractions for People With Visual Impairments and the Potential Role of On-Body Interaction ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Uran Oh , Lee Stearns , Alisha Pradhan , Jon E. Froehlich , and Leah Findlater . source | cite | search Augmented Reality Magnification for Low Vision Users With the Microsoft Hololens and a Finger-Worn Camera ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Lee Stearns , Victor DeSouza , Jessica Yin , Leah Findlater , and Jon E. Froehlich . source | cite | search Designing to Support and Extend the Competencies of People With Visual Impairments Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Gisela Reyes Cruz . source | cite | search Three-dimensional Nonvisual Directional Guidance for People With Visual Impairments 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). Seung A Chung , Kyungyeon Lee , Sohyeon Park , and Uran Oh . source | cite | search Social Sensemaking With AI: Designing an Open-ended AI Experience With a Blind Child Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Cecily Morrison , Edward Cutrell , Martin Grayson , Anja Thieme , Alex Taylor , Geert Roumen , Camilla Longden , Sebastian Tschiatschek , Rita Faia Marques , and Abigail Sellen . source | cite | search Digital Digits: Designing Assistive Finger Augmentation Devices Massachusetts Institute of Technology. Roy Shilkrot . source | cite | search Digital Digits: A Comprehensive Survey of Finger Augmentation Devices ACM Computing Surveys. Roy Shilkrot , Jochen Huber , Jürgen Steimle , Suranga Nanayakkara , and Pattie Maes . source | cite | search GestuRING: A Web-based Tool for Designing Gesture Input With Rings, Ring-Like, and Ring-Ready Devices 0. Radu-Daniel Vatavu and Laura-Bianca Bilius . source | cite | search TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users With Visual Impairments Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Lee Stearns , Uran Oh , Leah Findlater , and Jon E. Froehlich . source | cite | search Word Based Text Extraction Algorithm Implementation in Wearable Assistive Device for the Blind 2019 IEEE International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS). Arunima B Krishna , Meghana Hari , and Dr. Sudheer A.P . source | cite | search Understanding Needs, Identifying Opportunities: ICT in the View of Universal Design International Conference on Smart Objects and Technologies for Social Good. Ilaria Garofolo , Eric Medvet , Fulvio Babich , and Giovanni Ramponi . source | cite | search E-mail Management System for Blind People in Spanish Language Interacción '17: Proceedings of the XVIII International Conference on Human Computer Interaction. Guillermo Arturo Hernández Tapia and Ana Lilia Reyes-Herrera . source | cite | search AiSee: An Assistive Wearable Device to Support Visually Impaired Grocery Shoppers Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Roger Boldu , Denys J.C. Matthies , Haimo Zhang , and Suranga Nanayakkara . source | cite | search HearIt: Auditory-Cue-Based Audio Playback Control to Facilitate Information Browsing in Lecture Audio Applied Sciences. Jeongmin Hong , Hyesoo Jeon , Hana Lee , Daehyun Kim , and Minsam Ko . source | cite | search Printed Texts Tracking and Following for a Finger-Wearable Electro-Braille System Through Opto-electrotactile Feedback https://arxiv.org/abs/2109.02385.pdf. Mehdi Rahimi , Yantao Shen , Zhiming Liu , and Fang Jiang . source | cite | search FingerReader2.0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People With Visual Impairments While Shopping Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Roger Boldu , Alexandru Dancu , Denys J.C. Matthies , Thisum Buddhika , Shamane Siriwardhana , and Suranga Nanayakkara . source | cite | search Computer Vision Based Assistive Technology for Blind and Visually Impaired People ICCCNT '16: Proceedings of the 7th International Conference on Computing Communication and Networking Technologies. Shankar Sivan and Gopu Darsan . source | cite | search Localization of Skin Features on the Hand and Wrist From Small Image Patches 2016 23rd International Conference on Pattern Recognition (ICPR). Lee Stearns , Uran Oh , Bridget J. Cheng , Leah Findlater , David Ross , Rama Chellappa , and Jon E. Froehlich . source | cite | search Terminal Operation Learning Application for the Screen Reader Users 2015 IEEE International Conference on Systems, Man, and Cybernetics. Junji Onishi , Masatsugu Sakajiri , Takahiro Miura , and Tsukasa Ono . source | cite | search Interactive Technology for People With Colour Vision Deficiency Uppsala Universitet. Alexandra Kandler . source | cite | search Markit and Talkit Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. Lei Shi , Yuhang Zhao , and Shiri Azenkot . source | cite | search Handsight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments University of Maryland, College Park. Lee Stearns . source | cite | search Η Συμβολή Της Τεχνητής Νοημοσύνης Στην Εξ Αποστάσεως Εκπαίδευση Ατόμων Με Οπτική Αναπηρία Πανεπιστήμιο Μακεδονίας . Αναστάσιος Γκότσης . source | cite | search Diseño E Implementación De Un Sistema Electrónico De Lectura Para Estudiantes De Educación General Básica Con Discapacidad Visual UNIVERSIDAD NACIONAL DE CHIMBORAZO. Mireya Daniela Armendariz . source | cite | search HandSight: Supporting Everyday Activities Through Touch-Vision Annual Report, U.S. Army Medical Research and Materiel Command. Jon E. Froehlich , Leah Findlater , David Ross , and Rama Chellappa . source | cite | search FLight: A Low-Cost Reading and Writing System for Economically Less-Privileged Visually-Impaired People Exploiting Ink-based Braille System 6. Tusher Chakraborty , Taslim Arefin Khan , and A. B. M. Alim Al Islam . source | cite | search Towards Enabling Blind People to Independently Write on Printed Forms Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Shirin Feiz , Syed Masum Billah , Vikas Ashok , Roy Shilkrot , and IV Ramakrishnan . source | cite | search 
