Abstract
Free-form inpainting is the task of adding new content to an image in theregions specified by an arbitrary binary mask. Most existing approaches trainfor a certain distribution of masks, which limits their generalizationcapabilities to unseen mask types. Furthermore, training with pixel-wise andperceptual losses often leads to simple textural extensions towards the missingareas instead of semantically meaningful generation. In this work, we proposeRePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpaintingapproach that is applicable to even extreme masks. We employ a pretrainedunconditional DDPM as the generative prior. To condition the generationprocess, we only alter the reverse diffusion iterations by sampling theunmasked regions using the given image information. Since this technique doesnot modify or condition the original DDPM network itself, the model produceshigh-quality and diverse output images for any inpainting form. We validate ourmethod for both faces and general-purpose image inpainting using standard andextreme masks. RePaint outperforms state-of-the-art Autoregressive, and GAN approaches forat least five out of six mask distributions. Github Repository: git.io/RePaint