Practical Face Reconstruction via Differentiable Ray Tracing

  • 2021-01-13 21:36:11
  • Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cédric Thébault, Philippe-Henri Gosselin, Marco Romeo, Louis Chevallier
  • 40

Abstract

We present a differentiable ray-tracing based novel face reconstructionapproach where scene attributes - 3D geometry, reflectance (diffuse, specularand roughness), pose, camera parameters, and scene illumination - are estimatedfrom unconstrained monocular images. The proposed method models sceneillumination via a novel, parameterized virtual light stage, whichin-conjunction with differentiable ray-tracing, introduces a coarse-to-fineoptimization formulation for face reconstruction. Our method can not onlyhandle unconstrained illumination and self-shadows conditions, but alsoestimates diffuse and specular albedos. To estimate the face attributesconsistently and with practical semantics, a two-stage optimization strategysystematically uses a subset of parametric attributes, where subsequentattribute estimations factor those previously estimated. For example,self-shadows estimated during the first stage, later prevent its baking intothe personalized diffuse and specular albedos in the second stage. We show theefficacy of our approach in several real-world scenarios, where face attributescan be estimated even under extreme illumination conditions. Ablation studies,analyses and comparisons against several recent state-of-the-art methods showimproved accuracy and versatility of our approach. With consistent faceattributes reconstruction, our method leads to several style -- illumination,albedo, self-shadow -- edit and transfer applications, as discussed in thepaper.

 

Quick Read (beta)

loading the full paper ...