Abstract
Volumetric neural rendering methods, such as neural radiance fields (NeRFs),have enabled photo-realistic novel view synthesis. However, in their standardform, NeRFs do not support the editing of objects, such as a human head, withina scene. In this work, we propose RigNeRF, a system that goes beyond just novelview synthesis and enables full control of head pose and facial expressionslearned from a single portrait video. We model changes in head pose and facialexpressions using a deformation field that is guided by a 3D morphable facemodel (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns topredict only residuals to the 3DMM deformations and allows us to render novel(rigid) poses and (non-rigid) expressions that were not present in the inputsequence. Using only a smartphone-captured short video of a subject fortraining, we demonstrate the effectiveness of our method on free view synthesisof a portrait scene with explicit head pose and expression controls. Theproject page can be found here:http://shahrukhathar.github.io/2022/06/06/RigNeRF.html