PlenOctrees for Real-time Rendering of Neural Radiance Fields

  • 2021-03-25 17:59:06
  • Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa
  • 82

Abstract

We introduce a method to render Neural Radiance Fields (NeRFs) in real timeusing PlenOctrees, an octree-based 3D representation which supportsview-dependent effects. Our method can render 800x800 images at more than 150FPS, which is over 3000 times faster than conventional NeRFs. We do so withoutsacrificing quality while preserving the ability of NeRFs to performfree-viewpoint rendering of scenes with arbitrary geometry and view-dependenteffects. Real-time performance is achieved by pre-tabulating the NeRF into aPlenOctree. In order to preserve view-dependent effects such as specularities,we factorize the appearance via closed-form spherical basis functions.Specifically, we show that it is possible to train NeRFs to predict a sphericalharmonic representation of radiance, removing the viewing direction as an inputto the neural network. Furthermore, we show that PlenOctrees can be directlyoptimized to further minimize the reconstruction loss, which leads to equal orbetter quality compared to competing methods. Moreover, this octreeoptimization step can be used to reduce the training time, as we no longer needto wait for the NeRF training to converge fully. Our real-time neural renderingapproach may potentially enable new applications such as 6-DOF industrial andproduct visualizations, as well as next generation AR/VR systems. PlenOctreesare amenable to in-browser rendering as well; please visit the project page forthe interactive online demo, as well as video and code:https://alexyu.net/plenoctrees

 

Quick Read (beta)

loading the full paper ...