Abstract
Audio-Driven Talking Face Generation aims at generating realistic videos oftalking faces, focusing on accurate audio-lip synchronization withoutdeteriorating any identity-related visual details. Recent state-of-the-artmethods are based on inpainting, meaning that the lower half of the input faceis masked, and the model fills the masked region by generating lips alignedwith the given audio. Hence, to preserve identity-related visual details fromthe lower half, these approaches additionally require an unmasked identityreference image randomly selected from the same video. However, this commonmasking strategy suffers from (1) information loss in the input faces,significantly affecting the networks' ability to preserve visual quality andidentity details, (2) variation between identity reference and input imagedegrading reconstruction performance, and (3) the identity reference negativelyimpacting the model, causing unintended copying of elements unaligned with theaudio. To address these issues, we propose a mask-free talking face generationapproach while maintaining the 2D-based face editing task. Instead of maskingthe lower half, we transform the input images to have closed mouths, using atwo-step landmark-based approach trained in an unpaired manner. Subsequently,we provide these edited but unmasked faces to a lip adaptation model alongsidethe audio to generate appropriate lip movements. Thus, our approach needsneither masked input images nor identity reference images. We conductexperiments on the benchmark LRS2 and HDTF datasets and perform variousablation studies to validate our contributions.