Stitch it in Time: GAN-Based Facial Editing of Real Videos

  • 2022-01-20 18:48:20
  • Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H. Bermano, Daniel Cohen-Or
  • 53

Abstract

The ability of Generative Adversarial Networks to encode rich semanticswithin their latent space has been widely adopted for facial image editing.However, replicating their success with videos has proven challenging. Sets ofhigh-quality facial videos are lacking, and working with videos introduces afundamental barrier to overcome - temporal coherency. We propose that thisbarrier is largely artificial. The source video is already temporally coherent,and deviations from this state arise in part due to careless treatment ofindividual components in the editing pipeline. We leverage the naturalalignment of StyleGAN and the tendency of neural networks to learn lowfrequency functions, and demonstrate that they provide a strongly consistentprior. We draw on these insights and propose a framework for semantic editingof faces in videos, demonstrating significant improvements over the currentstate-of-the-art. Our method produces meaningful face manipulations, maintainsa higher degree of temporal consistency, and can be applied to challenging,high quality, talking head videos which current methods struggle with.

 

Quick Read (beta)

loading the full paper ...