As a Research Scientist at Google, I devote to inventing future interactive technologies in 3D graphics, fusing the information from the physical and virtual worlds, and making it interactive, accessible, and useful in VR, AR, and MR. I have published over 25 peer-reviewed publications in top venues of HCI, Computer Graphics, and Computer Vision, including CHI, SIGGRAPH Asia, UIST, TVCG, CVPR, ICCV, ECCV, ISMAR, VR, I3D, Web3D, etc. Please feel free to search keywords / authors / journal / conference below or visit my Google Scholar for more details.

Peer-reviewed Publications [bibTeX]

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video, slides, code, demo, supp | abstract | cite

Geollery: A Mixed Reality Social Media Platform

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world

pdf, doi | website, project, video, slides, demo | abstract | cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins

pdf, lowres, doi | website, project, video, slides | abstract | cite

Fusing Multimedia Data Into Dynamic Virtual Environments

Ruofei Du

Ph.D. Dissertation, Computer Science Department., University of Maryland, College Park., 2018.

Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world

pdf, lowres, doi | project, slides | abstract | cite

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

Keywords: accessibility, deaf, Deaf, hard of hearing, sound awareness

to appear | website, project, code | abstract | cite

GazeChat: Enhancing Virtual Conferences With Gaze-Aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation

pdf, doi | website, project, video, code | abstract | cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Haene, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

Keywords: deep implicit functions, neural representation, compression, levels of detail

pdf, lowres | project, code | abstract | cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

Keywords: correspondences, geodesic distance, embeddings, neural networks

pdf, lowres | website, project, video, code, demo, supp | abstract | cite

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table

pdf, doi | website, project, video, slides, code, data, supp | abstract | cite

Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP), 2021.

Keywords: deep learning, image compression

pdf, doi | project | abstract | cite

Saliency Computation for Virtual Cinematography in 360° Videos

Ruofei Du and Amitabh Varshney

IEEE Computer Graphics and Applications (CGA), 2021.

Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA

pdf, doi | website, project | abstract | cite

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication

pdf, doi | project, video, slides, code | abstract | cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization

pdf, lowres, doi | website, project, video, code, data | abstract | cite

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video | abstract | cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

Keywords: scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field

pdf, lowres, doi | project, video | abstract | cite

Eye-Dominance-Guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking

pdf, doi | project, video, code, supp | abstract | cite

Language-Based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization

pdf, lowres, doi | website, project, video, slides, code, data, supp | abstract | cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: GUI builder, layout manager, constraint-based layout, visual interface design, visual programming

pdf, doi | project, video, code | abstract | cite

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays

pdf, lowres, doi | website, project, video, slides, code, demo | abstract | cite

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world

pdf, lowres, doi | project, video, slides | abstract | cite

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cite

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang

European Conference on Computer Vision (ECCV), 2018.

Keywords: sketch dataset, scene sketch, sketch segmentation

pdf, lowres, doi | website, project, code, data, demo | abstract | cite

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, and Jon Froehlich

ACM Transactions on Accessible Computing (TACCESS), 2016.

Keywords: accessibility, real-time OCR, visual impairments, wearables

pdf, doi | website, project, video, slides | abstract | cite

Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment

Ruofei Du, Sujal Bista, and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR

pdf, lowres, doi | project, video, slides, data | abstract | cite

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.

Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world

pdf, doi | project, video | abstract | cite

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world

pdf, doi | project, video | abstract | cite

Tracking-Tolerent Visual Cryptography

Ruofei Du, Eric Lee, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: visual cryptography, augmented reality (AR), tracking

pdf, doi | project, video, data | abstract | cite

VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality

Ruofei Du and Liang He

Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.

Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display

pdf, lowres, doi | project, video, demo | abstract | cite

AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden

Ruofei Du, Kent Wills, Max Potasznik, and Jon Froehlich

Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.

Keywords: Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect

pdf, doi | website, project, video, code | abstract | cite

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon Froehlich

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.

Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind

pdf, doi | website, project, video, code | abstract | cite

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon Froehlich

Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.

Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch

pdf, doi | website, project, video, code | abstract | cite

Online Vigilance Analysis Combining Video and Electrooculography Features

Ruofei Du, Renjie Liu, Tianxiang Wu, and Bao-Liang Lu

Neural Information Processing - 19th International Conference (ICONIP), 2012.

Keywords: Vigilance Analysis Fatigue Detection Active Shape Model Electrooculography Support Vector Machine

pdf, doi | project, video | abstract | cite

Technical Reports

C-Flow: Visualizing Foot Traffic and Profit Data to Make Informative Decisions

Tiffany Chao, Ruofei Du, Jonathan Gluck, Hitesh Maidasani, Kent Wills, and Ben Shneiderman

University of Maryland, College Park. Department of Computer Science, 2012.

Keywords: Information Visualization; Data Mapping; Indoor Visualization; Business; Usability Testing

pdf, doi | project, video | abstract | cite

Statistics for K-Mer Based Splicing Analysis

Ruofei Du, Hao Li, Hui Miao, and Shangfu Peng

University of Maryland, College Park. Department of Computer Science, 2014.

Keywords:

pdf | project | abstract | cite

UistViz: 26 Years of UIST Coauthor Network Visualization

Ruofei Du

University of Maryland, College Park. Department of Computer Science, 2013.

Keywords: Information Visualization; UIST; Coauthor Network; DBLP; NodeXL

pdf | project | abstract | cite

Learning Depression Patterns From MyPersonality and Reddit

Weiwei Yang, Xuetong Sun, and Ruofei Du

University of Maryland, College Park. Department of Computer Science, 2015.

Keywords:

pdf | project | abstract | cite

Zero-Shot Learning Based Pedestrian Parsing

Xiyang Dai, Ruofei Du, and Hao Zhou

University of Maryland, College Park. Department of Computer Science, 2015.

Keywords: pedestrian parsing, zero-shot learning, segmentation

pdf | project | abstract | cite

Deliberately Planning and Acting for Angry Birds With Refinement Methods

Ruofei Du, Zebao Gao, and Zheng Xu

University of Maryland, College Park. Department of Computer Science, 2015.

Keywords:

pdf | project, video | abstract | cite

Interactive Mixed Reality Platform Utilizing Geotagged Social Media

Amitabh Varshney and Ruofei Du

US Patent 11,087,553, 2019.

Keywords:

pdf, lowres, url | project | abstract | cite

Fusing, Texturing, and Rendering Views of Dynamic Three-Dimensional Models

Ruofei Du, Ben Cutler, and Wayne Chang

US Patent 10,504,274, 2019.

Keywords:

pdf, lowres, url | project | abstract | cite

Systems, Devices, and Methods for Generating a Social Street View

Ruofei Du and Amitabh Varshney

US Patent 10,380,726, 2016.

Keywords: virtual reality; social media; street view; geographical information systems; mixed reality; WebGL

pdf, url | project | abstract | cite

• Navigate to Artsy
• Return to Home

Stay In Touch