The project "Improving Accessibility for Both Extended and Physical Reality" aims to improve access to digital and physical information for people with visual and hearing impairments. Project HandSight uses cameras and micro-haptic actuators on fingers to aid in reading and tactile exploration of physical and digital information. Project ProtoSound allows for customized sound recognition models on mobile devices for the deaf and hard of hearing.
VIDEO
Publications Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.
Keywords: accessibility, deaf, Deaf, hard of hearing, sound awareness
ACM Transactions on Accessible Computing (TACCESS), 2016.
Keywords: accessibility, real-time OCR, visual impairments, wearables, XR interaction
Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.
Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind
Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.
Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch
Cited By Understanding the Two-Step Nonvisual Omnidirectional Guidance for Target Acquisition in 3D Spaces 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). SeungA Chung , Kyungyeon Lee , and Uran Oh . source | cite | search Compact Microscopy Systems With Non-conventional Optical Techniques 9. Tahseen Kamal . source | cite | search Improving the Accessibility of Mobile OCR Apps Via Interactive Modalities ACM Transactions on Accessible Computing. Michael Cutter and Roberto Manduchi . source | cite | search Imagining Artificial Intelligence Applications With People With Visual Disabilities Using Tactile Ideation Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Cecily Morrison , Edward Cutrell , Anupama Dhareshwar , Kevin Doherty , Anja Thieme , and Alex Taylor . source | cite | search FingerReader: A Wearable Device to Explore Printed Text on the Go Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Roy Shilkrot , Jochen Huber , Wong Meng Ee , Pattie Maes , and Suranga Nanayakkara . source | cite | search FingerReader: A Finger-Worn Assistive Augmentation Assistive Augmentation. Roy Shilkrot , Jochen Huber , Roger Boldu , Pattie Maes , and Suranga Nanayakkara . source | cite | search Elastic Documents: Coupling Text and Tables Through Contextual Visualizations for Enhanced Document Reading IEEE Transactions on Visualization and Computer Graphics. Sriram Karthik Badam , Zhicheng Liu , and Niklas Elmqvist . source | cite | search Towards Automatic Diagram Description for the Blind Proceedings of the 11th International Convention on Rehabilitation Engineering and Assistive Technology. Seehait Chockthanyawat , Ekapol Chuangsuwanich , Atiwong Suchato , and Proadpran Punyabukkana . source | cite | search Computer Vision Methods for Visually Impaired People Fifteenth International Conference on Correlation Optics. Rostyslav Zatserkovnyi , Volodymyr Maik , and Roksoliana Zatserkovna . source | cite | search Opto-electrotactile Feedback Enabled Text-line Tracking Control for a Finger-wearable Reading Aid for the Blind 2022 International Conference on Robotics and Automation (ICRA). Mehdi Rahimi , Yantao Shen , Cong Peng , Zhiming Liu , and Fang Jiang . source | cite | search Computational Glasses: Repurposing Augmented Reality Glasses for Vision Assistance University of Otago. Jonathan Sutton . source | cite | search SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard of Hearing Users ASSETS '22: The 24th International ACM SIGACCESS Conference on Computers and Accessibility. Ziming Li , Shannon Connell , Shannon Connell , Wendy Dannels , Roshan Peiris , and Roshan Peiris . source | cite | search Stimulation of Numerical Skills in Children With Visual Impairments Using Image Recognition Procedia Computer Science. Jos{\'{e}} Antonio Taqu{\'{\i}}a Gutiérreza . source | cite | search Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement GI '16: Proceedings of the 42nd Graphics Interface Conference. Jonggi Hong , Lee Stearns , Jon E. Froehlich , David Ross , and Leah Findlater . source | cite | search Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Jonggi Hong , Alisha Pradhan , Jon E. Froehlich , and Leah Findlater . source | cite | search Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera 5. Alexander J. Medeiros , Lee Stearns , Leah Findlater , Chuan Chen , and Jon E. Froehlich . source | cite | search Towards a Natural User Interface to Support People With Visual Impairments in Detecting Colors 2016 23rd International Conference on Pattern Recognition (ICPR). Sergio Mascetti , Chiara Rossetti , Andrea Gerino , Cristian Bernareggi , Lorenzo Picinali , and Alessandro Rizzi . source | cite | search HapAR: Handy Intelligent Multimodal Haptic and Audio-Based Mobile AR Navigation for the Visually Impaired Technological Trends in Improved Mobility of the Visually Impaired. EAI/Springer Innovations in Communication and Computing. Ahmad Hoirul Basori . source | cite | search Investigating Microinteractions for People With Visual Impairments and the Potential Role of On-Body Interaction ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Uran Oh , Lee Stearns , Alisha Pradhan , Jon E. Froehlich , and Leah Findlater . source | cite | search Augmented Reality Magnification for Low Vision Users With the Microsoft Hololens and a Finger-Worn Camera ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. Lee Stearns , Victor DeSouza , Jessica Yin , Leah Findlater , and Jon E. Froehlich . source | cite | search Designing to Support and Extend the Competencies of People With Visual Impairments Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Gisela Reyes Cruz . source | cite | search Three-dimensional Nonvisual Directional Guidance for People With Visual Impairments 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). Seung A Chung , Kyungyeon Lee , Sohyeon Park , and Uran Oh . source | cite | search Social Sensemaking With AI: Designing an Open-ended AI Experience With a Blind Child Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Cecily Morrison , Edward Cutrell , Martin Grayson , Anja Thieme , Alex Taylor , Geert Roumen , Camilla Longden , Sebastian Tschiatschek , Rita Faia Marques , and Abigail Sellen . source | cite | search Digital Digits: Designing Assistive Finger Augmentation Devices Massachusetts Institute of Technology. Roy Shilkrot . source | cite | search Digital Digits: A Comprehensive Survey of Finger Augmentation Devices ACM Computing Surveys. Roy Shilkrot , Jochen Huber , Jürgen Steimle , Suranga Nanayakkara , and Pattie Maes . source | cite | search GestuRING: A Web-based Tool for Designing Gesture Input With Rings, Ring-Like, and Ring-Ready Devices 0. Radu-Daniel Vatavu and Laura-Bianca Bilius . source | cite | search TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users With Visual Impairments Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Lee Stearns , Uran Oh , Leah Findlater , and Jon E. Froehlich . source | cite | search Word Based Text Extraction Algorithm Implementation in Wearable Assistive Device for the Blind 2019 IEEE International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS). Arunima B Krishna , Meghana Hari , and Dr. Sudheer A.P . source | cite | search Understanding Needs, Identifying Opportunities: ICT in the View of Universal Design International Conference on Smart Objects and Technologies for Social Good. Ilaria Garofolo , Eric Medvet , Fulvio Babich , and Giovanni Ramponi . source | cite | search E-mail Management System for Blind People in Spanish Language Interacción '17: Proceedings of the XVIII International Conference on Human Computer Interaction. Guillermo Arturo Hernández Tapia and Ana Lilia Reyes-Herrera . source | cite | search AiSee: An Assistive Wearable Device to Support Visually Impaired Grocery Shoppers Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Roger Boldu , Denys J.C. Matthies , Haimo Zhang , and Suranga Nanayakkara . source | cite | search HearIt: Auditory-Cue-Based Audio Playback Control to Facilitate Information Browsing in Lecture Audio Applied Sciences. Jeongmin Hong , Hyesoo Jeon , Hana Lee , Daehyun Kim , and Minsam Ko . source | cite | search Printed Texts Tracking and Following for a Finger-Wearable Electro-Braille System Through Opto-electrotactile Feedback https://arxiv.org/abs/2109.02385.pdf. Mehdi Rahimi , Yantao Shen , Zhiming Liu , and Fang Jiang . source | cite | search FingerReader2.0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People With Visual Impairments While Shopping Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Roger Boldu , Alexandru Dancu , Denys J.C. Matthies , Thisum Buddhika , Shamane Siriwardhana , and Suranga Nanayakkara . source | cite | search Computer Vision Based Assistive Technology for Blind and Visually Impaired People ICCCNT '16: Proceedings of the 7th International Conference on Computing Communication and Networking Technologies. Shankar Sivan and Gopu Darsan . source | cite | search Localization of Skin Features on the Hand and Wrist From Small Image Patches 2016 23rd International Conference on Pattern Recognition (ICPR). Lee Stearns , Uran Oh , Bridget J. Cheng , Leah Findlater , David Ross , Rama Chellappa , and Jon E. Froehlich . source | cite | search Terminal Operation Learning Application for the Screen Reader Users 2015 IEEE International Conference on Systems, Man, and Cybernetics. Junji Onishi , Masatsugu Sakajiri , Takahiro Miura , and Tsukasa Ono . source | cite | search Interactive Technology for People With Colour Vision Deficiency Uppsala Universitet. Alexandra Kandler . source | cite | search Markit and Talkit Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. Lei Shi , Yuhang Zhao , and Shiri Azenkot . source | cite | search Handsight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments University of Maryland, College Park. Lee Stearns . source | cite | search Η Συμβολή Της Τεχνητής Νοημοσύνης Στην Εξ Αποστάσεως Εκπαίδευση Ατόμων Με Οπτική Αναπηρία Πανεπιστήμιο Μακεδονίας . Αναστάσιος Γκότσης . source | cite | search Diseño E Implementación De Un Sistema Electrónico De Lectura Para Estudiantes De Educación General Básica Con Discapacidad Visual UNIVERSIDAD NACIONAL DE CHIMBORAZO. Mireya Daniela Armendariz . source | cite | search HandSight: Supporting Everyday Activities Through Touch-Vision Annual Report, U.S. Army Medical Research and Materiel Command. Jon E. Froehlich , Leah Findlater , David Ross , and Rama Chellappa . source | cite | search FLight: A Low-Cost Reading and Writing System for Economically Less-Privileged Visually-Impaired People Exploiting Ink-based Braille System 6. Tusher Chakraborty , Taslim Arefin Khan , and A. B. M. Alim Al Islam . source | cite | search Towards Enabling Blind People to Independently Write on Printed Forms Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Shirin Feiz , Syed Masum Billah , Vikas Ashok , Roy Shilkrot , and IV Ramakrishnan . source | cite | search HiSSNet: Sound Event Detection and Speaker Identification Via Hierarchical Prototypical Networks for Low-Resource Headphones arXiv.2303.07538. N Shashaank , Berker Banar , Mohammad Rasool Izadi , Jeremy Kemmerer , Shuo Zhang , and Chuan-Che Huang . source | cite | search A Multimodal Prototypical Approach for Unsupervised Sound Classification arXiv.2306.12300. Saksham Singh Kushwaha and Magdalena Fuentes . source | cite | search Silent Delivery: Make Instant Delivery More Accessible for the DHH Delivery Workers Through Sensory Substitution Distributed, Ambient and Pervasive Interactions. Shichao Huang , Xiaolong Li , Shang Shi , Haoye Dong , Xueyan Cai , Kecheng Jin , Jiayi Wu , Weijia Lin , Jiayu Yao , Yuqi Hu , Fangtian Ying , and Cheng Yao . source | cite | search Socio-Technical Trust for Multi-Modal Hearing Assistive Technology 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). Jennifer Williams , Tayyaba Azim , Anna-Maria Piskopani , Alan Chamberlain , and Shuo Zhang . source | cite | search A Survey on Artificial Intelligence-Based Acoustic Source Identification IEEE Access. Ruba Zaheer , Iftekhar Ahmad , Daryoush Habibi , Kazi Yasin Islam , and Quoc Viet Phung . source | cite | search 
