Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition

  • 2022-11-22 16:03:11
  • Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, Jingdong Wang
  • 198

Abstract

While dynamic Neural Radiance Fields (NeRF) have shown success inhigh-fidelity 3D modeling of talking portraits, the slow training and inferencespeed severely obstruct their potential usage. In this paper, we propose anefficient NeRF-based framework that enables real-time synthesizing of talkingportraits and faster convergence by leveraging the recent success of grid-basedNeRF. Our key insight is to decompose the inherently high-dimensional talkingportrait representation into three low-dimensional feature grids. Specifically,a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3Dspatial grid and a 2D audio grid. The torso is handled with another 2D grid ina lightweight Pseudo-3D Deformable Module. Both modules focus on efficiencyunder the premise of good rendering quality. Extensive experiments demonstratethat our method can generate realistic and audio-lips synchronized talkingportrait videos, while also being highly efficient compared to previousmethods.

 

Quick Read (beta)

loading the full paper ...