DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Real-time depth data is readily available on mobile phones with passive or active sensors and on VR/AR devices. However, this rich data about our environment is under-explored in mainstream AR applications. Slow adoption of depth information in the UX layer may be due to the complexity of processing depth data to simply render a mesh or detect interaction based on changes in the depth map. In this paper, we introduce DepthLab, a software library that encapsulates a variety of UI/UX features for depth, including geometry-aware rendering (occlusion, shadows), depth interactive behaviors (physically based collisions, avatar path planning), and visual effects (relighting, aperture effects). We break down the usage of depth map into point depth, surface depth, and per-pixel depth and introduce our real-time algorithms for different use cases. We present the design process, system, and components of DepthLab to streamline and centralize the development of interactive depth features. We open-sourced our software with external developers and collected both qualitative and quantitative feedback. Our results and feedback from engineers suggest that DepthLab help mobile AR developers unleash their creativity and effortlessly integrate depth in mobile AR experiences, and amplify their prototyping efforts.

Publications

teaser image of DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore



teaser image of Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore



Videos

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality


[UIST 2020 5-min Talk] DepthLab


AR Realism with the ARCore Depth API


Blending realities with the ARCore Depth API - Deep Dive


Talks

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality Teaser Image.

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du

UIST 2020, Online Talk.


Fusing Physical and Virtual Worlds into 
Interactive Mixed Reality Teaser Image.

Fusing Physical and Virtual Worlds into Interactive Mixed Reality

Ruofei Du

GMU Invited Talk , Online Talk.


Cited By

  • Neural Cameras: Learning Camera Characteristics for Coherent Mixed Reality Rendering. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR. David Mandl, Peter Mohr, Tobias Langlotz, Christoph Ebner, Shohei Mori, Stefanie Zollmann, Peter M Roth, and Denis Kalkofen. [doi]
  • Lines of Play: Using Vector Tools to Place Game Objects in Augmented Reality. Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play. Jasmine Roberts, John Mernacaj, Suzanne Chambers, Dror Ayalon, and Errol King. [doi]
  • ARToken: A Tangible Device for Dynamically Binding Real-World Objects With Virtual Representation. Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. Hsuan-Yu Hsueh, Chien-Hua Chen, Irene Chen, Chih-Yuan Yao, and Hung-Kuo Chu. [doi]
  • VRFromX: From Scanned Reality to Interactive Virtual Experience With Human-in-the-Loop. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Ananya Ipsita, Hao Li, Runlin Duan, Yuanzhi Cao, Subramanian Chidambaram, Min Liu, and Karthik Ramani. [doi]
  • Binary Descriptors for Dense Stereo Matching. International Journal of Intelligent Computing and Information Sciences. Hanaa Ibrahim, Heba Khaled, Noha AbdElSabour Seada, and Hossam Faheem. [doi]
  • Snap2cad: 3D Indoor Environment Reconstruction for AR/VR Applications Using a Smartphone Device. Computers \& Graphics. Alessandro Manni, Damiano Oriti, Andrea Sanna, Francesco De Pace, and Federico Manuri. [doi]
  • Portalware: Exploring Free-Hand AR Drawing With a Dual-Display Smartphone-Wearable Paradigm. Designing Interactive Systems Conference 2021.Jing Qian, Tongyu Zhou, Meredith Young-Ng, Jiaju Ma, Angel Cheung, Xiangyu Li, Ian Gonsher, and Jeff Huang. [doi]
  • Measuring Forest Carbon With Mobile Phones. Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. Amelia Holcomb, Bill Tong, Megan Penny, and Srinivasan Keshav. [doi]
  • Enabling Real-Time Sign Language Translation on Mobile Platforms With On-Board Depth Cameras. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. HyeonJung Park, Youngki Lee, and JeongGil Ko. [doi]
  • Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence. https://arxiv.org/abs/2106.08710. Jacky Cao, Kit-Yung Lam, Lik-Hang Lee, Xiaoli Liu, Pan Hui, and Xiang Su. [doi]
  • DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Zeyu Wang, Cuong Nguyen, Paul Asente, and Julie Dorsey. [doi]
  • Semantic Mapping: A Semantics-Based Approach to Virtual Content Placement for Immersive Environments. 2021 17th International Conference on Intelligent Environments (IE). Jingyang Liu. [doi]
  • Validation of Robot Model With Mobile Augmented Reality. 2021 6th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS). Chanapol Piyavichayanon and Masanobu Koga. [doi]
  • Outdoor Objekt-Detektion Und -Tracking Von Verkehrsteilnehmern Für Augmented Reality. Fakultät Technik und Informatik. Ruben Horn. [doi]
  • Landscape Visualization by Integrating Augmented Reality and Drones With Occlusion Handling to Link Real and Virtual Worlds - Towards City Digital Twin Realization. . Kikuch Kikuchi, Naok Naoki, Fukud Fukuda, Tomohir Tomohiro, Yabuk Yabuki, and Nobuyosh Nobuyoshi. [doi]
  • AdaBins: Depth Estimation Using Adaptive Bins. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. [doi]
  • Stay In Touch