Abstract
We present a fully automatic system that can produce high-fidelity,photo-realistic 3D digital human characters with a consumer RGB-D selfiecamera. The system only needs the user to take a short selfie RGB-D video whilerotating his/her head, and can produce a high quality reconstruction in lessthan 30 seconds. Our main contribution is a new facial geometry modeling andreflectance synthesis procedure that significantly improves thestate-of-the-art. Specifically, given the input video a two-stage frameselection algorithm is first employed to select a few high-quality frames forreconstruction. A novel, differentiable renderer based 3D Morphable Model(3DMM) fitting method is then applied to recover facial geometries frommultiview RGB-D data, which takes advantages of extensive data generation andperturbation. Our 3DMM has much larger expressive capacities than conventional3DMM, allowing us to recover more accurate facial geometry using merely linearbases. For reflectance synthesis, we present a hybrid approach that combinesparametric fitting and CNNs to synthesize high-resolution albedo/normal mapswith realistic hair/pore/wrinkle details. Results show that our system canproduce faithful 3D characters with extremely realistic details. Code and theconstructed 3DMM is publicly available.