HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video

  • 2022-01-11 18:51:21
  • Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
  • 20

Abstract

We introduce a free-viewpoint rendering method -- HumanNeRF -- that works ona given monocular video of a human performing complex body motions, e.g. avideo from YouTube. Our method enables pausing the video at any frame andrendering the subject from arbitrary new camera viewpoints or even a full360-degree camera path for that particular frame and body pose. This task isparticularly challenging, as it requires synthesizing photorealistic details ofthe body, as seen from various camera angles that may not exist in the inputvideo, as well as synthesizing fine details such as cloth folds and facialappearance. Our method optimizes for a volumetric representation of the personin a canonical T-pose, in concert with a motion field that maps the estimatedcanonical representation to every frame of the video via backward warps. Themotion field is decomposed into skeletal rigid and non-rigid motions, producedby deep networks. We show significant performance improvements over prior work,and compelling examples of free-viewpoint renderings from monocular video ofmoving humans in challenging uncontrolled capture scenarios.

 

Quick Read (beta)

loading the full paper ...