Extracting Triangular 3D Models, Materials, and Lighting From Images

  • 2021-11-24 13:58:20
  • Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, Sanja Fidler
  • 43

Abstract

We present an efficient method for joint optimization of topology, materialsand lighting from multi-view image observations. Unlike recent multi-viewreconstruction approaches, which typically produce entangled 3D representationsencoded in neural networks, we output triangle meshes with spatially-varyingmaterials and environment lighting that can be deployed in any traditionalgraphics engine unmodified. We leverage recent work in differentiablerendering, coordinate-based networks to compactly represent volumetrictexturing, alongside differentiable marching tetrahedrons to enablegradient-based optimization directly on the surface mesh. Finally, we introducea differentiable formulation of the split sum approximation of environmentlighting to efficiently recover all-frequency lighting. Experiments show ourextracted models used in advanced scene editing, material decomposition, andhigh quality view interpolation, all running at interactive rates intriangle-based renderers (rasterizers and path tracers).

 

Quick Read (beta)

loading the full paper ...