Abstract
Rendering is the process of generating 2D images from 3D assets, simulated ina virtual environment, typically with a graphics pipeline. By inverting suchrenderer, one can think of a learning approach to predict a 3D shape from aninput image. However, standard rendering pipelines involve a fundamentaldiscretization step called rasterization, which prevents the rendering processto be differentiable, hence suitable for learning. We present the firstnon-parametric and truly differentiable rasterizer based on silhouettes. Ourmethod enables unsupervised learning for high-quality 3D mesh reconstructionfrom a single image. We call our framework `soft rasterizer' as it provides anaccurate soft approximation of the standard rasterizer. The key idea is to fusethe probabilistic contributions of all mesh triangles with respect to therendered pixels. When combined with a mesh generator in a deep neural network,our soft rasterizer is able to generate an approximated silhouette of thegenerated polygon mesh in the forward pass. The rendering loss isback-propagated to supervise the mesh generation without the need of 3Dtraining data. Experimental results demonstrate that our approach significantlyoutperforms the state-of-the-art unsupervised techniques, both quantitativelyand qualitatively. We also show that our soft rasterizer can achieve comparableresults to the cutting-edge supervised learning method and in various caseseven better ones, especially for real-world data.