VToonify: Controllable High-Resolution Portrait Video Style Transfer

  • 2022-09-22 18:59:10
  • Shuai Yang, Liming Jiang, Ziwei Liu, Chen Change Loy
  • 174

Abstract

Generating high-quality artistic portrait videos is an important anddesirable task in computer graphics and vision. Although a series of successfulportrait image toonification models built upon the powerful StyleGAN have beenproposed, these image-oriented methods have obvious limitations when applied tovideos, such as the fixed frame size, the requirement of face alignment,missing non-facial details and temporal inconsistency. In this work, weinvestigate the challenging controllable high-resolution portrait video styletransfer by introducing a novel VToonify framework. Specifically, VToonifyleverages the mid- and high-resolution layers of StyleGAN to renderhigh-quality artistic portraits based on the multi-scale content featuresextracted by an encoder to better preserve the frame details. The resultingfully convolutional architecture accepts non-aligned faces in videos ofvariable size as input, contributing to complete face regions with naturalmotions in the output. Our framework is compatible with existing StyleGAN-basedimage toonification models to extend them to video toonification, and inheritsappealing features of these models for flexible style control on color andintensity. This work presents two instantiations of VToonify built upon Toonifyand DualStyleGAN for collection-based and exemplar-based portrait video styletransfer, respectively. Extensive experimental results demonstrate theeffectiveness of our proposed VToonify framework over existing methods ingenerating high-quality and temporally-coherent artistic portrait videos withflexible style controls.

 

Quick Read (beta)

loading the full paper ...