Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this paper, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47\times-7.28\times speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.
VIDEO
Publications IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table, eye tracking, digital world
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization, eye tracking, interactive graphics
IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.
Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking, interactive graphics
Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.
Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays, eye tracking, interactive graphics
Cited By Foveated Real-Time Path Tracing in Visual-Polar Space Eurographics Symposium on Rendering - DL-only and Industry Track. Matias Koskela , Atro Lotvonen , Markku Mäkitalo , Petrus Kivi , Timo Viitanen , and Pekka Jääskeläinen . source | cite | search Perception-Driven Hybrid Foveated Depth of Field Rendering for Head-Mounted Displays 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Jingyu Liu , Claire Mantel , and Soren Forchhammer . source | cite | search Pistol: Pupil Invisible Supportive Tool to Extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D As Well As 3D Gaze https://arxiv.org/abs/2201.06799. Wolfgang Fuhl , Daniel Weber , and Enkelejda Kasneci . source | cite | search Foveated Depth-of-Field Filtering in Head-Mounted Displays ACM Transactions on Applied Perception. Martin Weier , Thorsten Roth , André Hinkenjann , and Philipp Slusallek . source | cite | search Foveated Streaming of Real-time Graphics Proceedings of the 12th ACM Multimedia Systems Conference. Gazi Karam Illahi , Matti Siekkinen , Teemu Kämäräinen , and Antti Ylä-Jääski . source | cite | search Selective Foveated Ray Tracing for Head-Mounted Displays 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Youngwook Kim , Yunmin Ko , and Insung Ihm . source | cite | search Noise-based Enhancement for Foveated Rendering ACM Transactions on Graphics. Taimoor Tariq , Cara Tursun , and Piotr Didyk . source | cite | search Rectangular Mapping-based Foveated Rendering The IEEE Conference on Virtual Reality and 3D User Interfaces 2022 (IEEE VR). Jiannan Ye , Anqi Xie , Susmjia Jabbireddy , Yunchuan Li , Xubo Yang , and Xiaoxu Meng . source | cite | search An Integrative View of Foveated Rendering 0. Bipul Mohanto , ABM Tariqul Islam , Enrico Gobbetti , and Oliver Staadt . source | cite | search A Simulation System for Scene Synthesis in Virtual Reality Virtual Reality and Mixed Reality. Jingyu Liu , Claire Mantel , Florian Schweiger , and S{\o}ren Forchhammer . source | cite | search A Cybersickness Review: Causes, Strategies, and Classification Methods Journal on Interactive Systems. Thiago Porcino , Daniela Trevisan , and Esteban Clua . source | cite | search Building Virtual Scene Construction and Environmental Impact Analysis Based on Image Processing Scientific Programming. Fan Ying and Zhou Bo . source | cite | search Dynamic Viewport Selection-Based Prioritized Bitrate Adaptation for Tile-Based 360\textdegree Video Streaming IEEE Access. Abid Yaqoob , Mohammed Amine Togou , and Gabriel-Miro Muntean . source | cite | search Practical Saccade Prediction for Head-Mounted Displays: Towards a Comprehensive Model ACM Transactions on Applied Perception. Elena Arabadzhiyska , Cara Tursun , Hans-Peter Seidel , and Piotr Didyk . source | cite | search Foveated Rendering: Motivation, Taxonomy, and Research Directions arXiv.2205.04529. Susmija Jabbireddy , Xuetong Sun , Xiaoxu Meng , and Amitabh Varshney . source | cite | search A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry Proceedings of the ACM on Human-Computer Interaction. Johannes Meyer , Thomas Schlebusch , and Enkelejda Kasneci . source | cite | search Visual Perception in Virtual Reality and the Application in Sports Universität Magdeburg. Stefan Pastel . source | cite | search Modeling and Optimizing Human-in-the-Loop Visual Perception SID Symposium Digest of Technical Papers. Qi Sun , Budmonde Duinkharjav , and Anjul Patney . source | cite | search Deep Light Field Spatial Super-Resolution Using Heterogeneous Imaging IEEE Transactions on Visualization and Computer Graphics. Yeyao Chen , Gangyi Jiang , Mei Yu , Haiyong Xu , and Yo-Sung Ho . source | cite | search A Real 3D Scene Rendering Optimization Method Based on Region of Interest and Viewing Frustum Prediction in Virtual Reality International Journal of Digital Earth. Pei Dang , Jun Zhu , Jianlin Wu , Weilian Li , Jigang You , Lin Fu , Yiqun Shi , and Yuhang Gong . source | cite | search Application of Eye-tracking Systems Integrated Into Immersive Virtual Reality and Possible Transfer to the Sports Sector - a Systematic Review Multimedia Tools and Applications. Stefan Pastel , Josua Marlok , Nicole Bandow , and Kerstin Witte . source | cite | search Progressive Multi-scale Light Field Networks arXiv.2208.06710. David Li and Amitabh Varshney . source | cite | search Wavelet-Based Fast Decoding of 360° Videos arXiv.2208.10859. Colin Groth , Sascha Fricke , Susana Castillo , and Marcus Magnor . source | cite | search Bullet Comments for 360\textdegreeVideo 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Yi-Jun Li , Jinchuan Shi , Fang-Lue Zhang , and Miao Wang . source | cite | search Look Over There! Investigating Saliency Modulation for Visual Guidance With Augmented Reality Glasses The 35th Annual ACM Symposium on User Interface Software and Technology. Jonathan Sutton , Tobias Langlotz , Alexander Plopski , Stefanie Zollmann , Yuta Itoh , and Holger Regenbrecht . source | cite | search Rega-Net:Retina Gabor Attention for Deep Convolutional Neural Networks arXiv.2211.12698. Chun Bao , Jie Cao , Yaqian Ning , Yang Cheng , and Qun Hao . source | cite | search Foveated Rendering: A State-of-the-Art Survey arXiv.2211.07969. Lili Wang , Xuehuai Shi , and Yi Liu . source | cite | search Towards Energy Efficient Mobile Eye Tracking for AR Glasses Through Optical Sensor Technology arXiv.2212.03189. Johannes Meyer . source | cite | search Beyond Flicker, Beyond Blur: View-coherent Metameric Light Fields for Foveated Display SIGGRAPH Posters. Prithvi Kohli , David R. Walton , Rafael Kuffner dos Anjos , Anthony Steed , and Tobias Ritschel . source | cite | search Foveated Rendering Technique in Modern Graphics University of Maryland, College Park. Xiaoxu Meng . source | cite | search Time-Warped Foveated Rendering for Virtual Reality Headsets Computer Graphics Forum. Linus Franke , Laura Fink , Jana Martschinke , Kai Selgrad , and Marc Stamminger . source | cite | search Foveated Instant Radiosity 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Lili Wang , Runze Li , Xuehuai Shi , Ling-Qi Yan , and Zhichao Li . source | cite | search Foveated Light Culling Computers & Graphics. Qinqi Yang , Zhuxin Chen , Yanli Liu , Guanyu Xing , and Yanci Zhang . source | cite | search A Unified Deep Learning Approach for Foveated Rendering \& Novel View Synthesis From Sparse RGB-D Light Fields 2020 International Conference on 3D Immersion (IC3D). Vineet Thumuluri and Mansi Sharma . source | cite | search On the Interplay of Foveated Rendering and Video Encoding 26th ACM Symposium on Virtual Reality Software and Technology. Gazi Karam Illahi , Matti Siekkinen , Teemu Kämäräinen , and Antti Ylä-Jääski . source | cite | search Modelling Foveated Depth-of-field Blur for Improving Depth Perception in Virtual Reality 2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS). Razeen Hussain , Manuela Chessa , and Fabio Solari . source | cite | search Toward Standardized Classification of Foveated Displays IEEE Transactions on Visualization and Computer Graphics. Josef Spjut , Ben Boudaoud , Jonghyun Kim , Trey Greer , Rachel Albert , Michael Stengel , Kaan Aksit , and David Luebke . source | cite | search Learning Foveated Reconstruction to Preserve Perceived Image Statistics https://arxiv.org/abs/2108.03499. Luca Surace , Marek Wernikowski , Okan Tursun , Karol Myszkowski , Rados{\l}aw Mantiuk , and Piotr Didyk . source | cite | search State of the Art in Perceptual VR Displays Real VR –Immersive Digital Reality. Gordon Wetzstein , Anjul Patney , and Qi Sun . source | cite | search How the Presence and Size of Static Peripheral Blur Affects Cybersickness in Virtual Reality ACM Transactions on Applied Perception. Yun-Xuan Lin , Rohith Venkatakrishnan , Roshan Venkatakrishnan , Elham Ebrahimi , Wen-Chieh Lin , and Sabarish V. Babu . source | cite | search Cybersickness Analysis Using Symbolic Machine Learning Algorithms Universidade Federal Fluminense. Thiago Porcina . source | cite | search Eye Tracking Data Collection Protocol for VR for Remotely Located Subjects Using Blockchain and Smart Contracts 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). Efe Bozkir , Shahram Eivazi , Mete Akgun , and Enkelejda Kasneci . source | cite | search A Privacy-preserving Approach to Streaming Eye-tracking Data IEEE Transactions on Visualization and Computer Graphics. Brendan David-John , Diane Hosfelt , Kevin Butler , and Eakta Jain . source | cite | search FAVR - Accelerating Direct Volume Rendering for Virtual RealitySystems 2020 IEEE Visualization Conference (VIS). Andre Waschk and Jens Kruger . source | cite | search Focus Guided Light Field Saliency Estimation 2021 13th International Conference on Quality of Multimedia Experience (QoMEX). Ailbhe Gill , Emin Zerman , Martin Alain , Mikael Le Pendu , and Aljosa Smolic . source | cite | search Foveated Photon Mapping IEEE Transactions on Visualization and Computer Graphics. Xuehuai Shi , Lili Wang , Xiaoheng Wei , and Ling-Qi Yan . source | cite | search Exploiting Object-of-Interest Information to Understand Attention in VR Classrooms 2021 IEEE Virtual Reality and 3D User Interfaces (VR). Efe Bozkir , Philipp Stark , Hong Gao , Lisa Hasenbein , Jens-Uwe Hahn , Enkelejda Kasneci , and Richard Gollner . source | cite | search Rendering Optimizations for Virtual Reality Using Eye-Tracking 2020 22nd Symposium on Virtual and Augmented Reality (SVR). Shawn Matthews , Alvaro Uribe-Quevedo , and Alexander Theodorou . source | cite | search Towards Everyday Virtual Reality Through Eye Tracking der Eberhard Karls Universität Tübingen. Efe Bozkir . source | cite | search HPCGen: Hierarchical K-Means Clustering and Level Based Principal Components for Scan Path Genaration . Wolfgang Fuhl . source | cite | search Perceptual Rasterization for Head-mounted Display Image Synthesis ACM Transactions on Graphics. Sebastian Friston , Tobias Ritschel , and Anthony Steed . source | cite | search A Review of Resolution Losses for AR/VR Foveated Imaging Applications 2018 IEEE Games, Entertainment, Media Conference (GEM). Timothee E. Cognard , Alexander Goncharov , Nicholas Devaney , Chris Dainty , and Peter Corcoran . source | cite | search Impact of Foveated Rendering on Procedural Task Training Virtual, Augmented and Mixed Reality. Multimodal Interaction. Rafael Radkowski and Supriya Raul . source | cite | search Masked360: Enabling Robust 360-degree Video Streaming With Ultra Low Bandwidth Consumption IEEE Transactions on Visualization and Computer Graphics. Zhenxiao Luo , Baili Chai , Zelong Wang , Miao Hu , and Di Wu . source | cite | search Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges arXiv.2305.14080. Efe Bozkir , Süleyman Özdel , Mengdi Wang , Brendan David-John , Hong Gao , Kevin Butler , Eakta Jain , and Enkelejda Kasneci . source | cite | search LFACon: Introducing Anglewise Attention to No-Reference Quality Assessment in Light Field Space IEEE Transactions on Visualization and Computer Graphics. Qiang Qu , Xiaoming Chen , Yuk Ying Chung , and Weidong Cai . source | cite | search Quality-Constrained Encoding Optimization for Omnidirectional Video Streaming IEEE Transactions on Circuits and Systems for Video Technology. Chaofan He , Roberto G. de A. Azevedo , Jiacheng Chen , Shuyuan Zhu , Bing Zeng , and Pascal Frossard . source | cite | search Providing Privacy for Eye-Tracking Data With Applications in XR University of Florida. Brendan David-John . source | cite | search Polar Coordinate Fourier Single-pixel Imaging Optics Letters. Guan Wang , Huaxia Deng , Mengchao Ma , and Xiang Zhong . source | cite | search Foveated Rendering: A State-of-the-art Survey Computational Visual Media. Lili Wang , Xuehuai Shi , and Yi Liu . source | cite | search Data and Model Hybrid-driven Virtual Reality Robot Operating System Frontiers in Energy Research. Xinyu Liu , Lin Nan , Yuexin Lin , Jiatong Han , Jinxin Liu , and Tao Ku . source | cite | search Area of Interest Adaption Using Feature Importance arXiv.2303.12744. Wolfgang Fuhl , Susanne Zabel , Theresa Harbig , Julia Astrid Moldt , Teresa Festl Wiete , Anne Herrmann Werner , and Kay Nieselt . source | cite | search Software-Defined Imaging: A Survey Proceedings of the IEEE. Suren Jayasuriya , Odrika Iqbal , Venkatesh Kodukula , Victor Torres , Robert LiKamWa , and Andreas Spanias . source | cite | search Locomotion-aware Foveated Rendering 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). Xuehuai Shi , Lili Wang , Jian Wu , Wei Ke , and Chan-Tong Lam . source | cite | search Design Guidelines for Limiting and Eliminating Virtual Reality-induced Symptoms and Effects at Work: A Comprehensive, Factor-oriented Review Frontiers in Psychology. Alexis D. Souchet , Domitile Lourdeaux , Jean-Marie Burkhardt , and Peter A. Hancock . source | cite | search Non-homogeneous Denoising for Virtual Reality in Real-time Path Tracing Rendering Graphical Models. Victor Peres , Esteban Clua , Thiago Porcino , and Anselmo Montenegro . source | cite | search Compact Near-eye Display for Firefighter\textquotesingles Self-contained Breathing Apparatus ETRI Journal. Ungyeon Yang . source | cite | search Individualized Foveated Rendering With Eye-tracking Head-mounted Display Virtual Reality. Jihwan Kim , Jejoong Kim , Myeongul Jung , Taesoo Kwon , and Kwanguk Kenny Kim . source | cite | search Exploring Foveation Techniques for Virtual Reality Environments Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. Razeen Hussain , Manuela Chessa , and Fabio Solari . source | cite | search Neural Foveated Super-Resolution for Real-time VR Rendering 1. Jiannan Ye , Xiaoxu Meng , Daiyun Guo , Cheng Shang , Haotian Mao , and Xubo Yang . source | cite | search Scene-aware Foveated Rendering IEEE Transactions on Visualization and Computer Graphics. Runze Fan , Xuehuai Shi , Kangyu Wang , Qixiang Ma , and Lili Wang . source | cite | search Virtual Reality Telepresence: 360-Degree Video Streaming With Edge-Compute Assisted Static Foveated Compression IEEE Transactions on Visualization and Computer Graphics. Xincheng Huang , James Riddell , and Robert Xiao . source | cite | search 
