Deep 3D Representations

We introduce a new implicit shape representation called Primary Ray-based Implicit Function (PRIF). In contrast to most existing approaches based on the signed distance function (SDF) which handles spatial locations, our representation operates on oriented rays. Specifically, PRIF is formulated to directly produce the surface hit point of a given input ray, without the expensive sphere-tracing operations, hence enabling efficient shape extraction and differentiable rendering. We demonstrate that neural networks trained to encode PRIF achieve successes in various tasks including single shape representation, category-wise shape generation, shape completion from sparse or noisy observations, inverse rendering for camera pose estimation, and neural rendering with color.
We introduce Multiresolution Deep Implicit Functions (MDIF), a hierarchical representation that can recover fine geometry detail, while being able to perform global operations such as shape completion. Our model represents a complex 3D shape with a hierarchy of latent grids, which can be decoded into different levels of detail and also achieve better accuracy. For shape completion, we propose latent grid dropout to simulate partial data in the latent space and therefore defer the completing functionality to the decoder side. This along with our multires design significantly improves the shape completion quality under decoder-only latent optimization. To the best of our knowledge, MDIF is the first deep implicit function model that can at the same time (1) represent different levels of detail and allow progressive decoding; (2) support both encoder-decoder inference and decoder-only latent optimization, and fulfill multiple applications; (3) perform detailed decoder-only shape completion. Experiments demonstrate its superior performance against prior art in various 3D reconstruction tasks.

Publications

teaser image of PRIF: Primary Ray-Based Implicit Function

PRIF: Primary Ray-Based Implicit Function

European Conference on Computer Vision (ECCV), 2022.
Keywords: deep implicit functions, neural representation, signed distance function
teaser image of Multiresolution Deep Implicit Functions for 3D Shape Representation

Multiresolution Deep Implicit Functions for 3D Shape Representation

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Keywords: deep implicit functions, neural representation, compression, levels of detail

Videos

Talks

Cited By

  • BACON: Band-Limited Coordinate Networks for Multiscale Scene Representation. https://arxiv.org/pdf/2112.04645.pdf. David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. source | cite
  • UNIST: Unpaired Neural Implicit Shape Translation Network. https://arxiv.org/pdf/2112.05381.pdf. Qimin Chen, Johannes Merz, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amiri, and Hao Zhang. source | cite
  • PINs: Progressive Implicit Networks for Multi-Scale Neural Representations. https://arxiv.org/pdf/2202.04713.pdf. Zoe Landgraf, Alexander Sorkine Hornung, and Ricardo Silveira Cabral. source | cite
  • Learning Deep Implicit Functions for 3D Shapes With Dynamic Code Clouds. https://arxiv.org/abs/2203.14048. Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. source | cite
  • Towards Implicit Text-Guided 3D Shape Generation. https://arxiv.org/abs/2203.14622. Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. source | cite
  • Supplementary for “Learning Deep Implicit Functions for 3D Shapes With Dynamic Code Clouds”. https://arxiv.org/abs/2203.14048. Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. source | cite
  • Neural Fields in Visual Computing and Beyond. arXiv.2111.11426. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. source | cite
  • 3DILG: Irregular Latent Grids for 3D Generative Modeling. arXiv.2205.13914. Biao Zhang, Matthias Nießner, and Peter Wonka. source | cite
  • PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories. arXiv.2206.04916. Yuchen Rao, Yinyu Nie, and Angela Dai. source | cite
  • Preprocessing Enhanced Image Compression for Machine Vision. arXiv.2206.05650. Guo Lu, Xingtong Ge, Tianxiong Zhong, Jing Geng, and Qiang Hu. source | cite
  • PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation. arXiv.2207.11790. Bo Sun, Vladimir G. Kim, Noam Aigerman, Qixing Huang, and Siddhartha Chaudhuri. source | cite
  • Stay In Touch