Abstract
Denoising diffusion probabilistic models for image inpainting aim to add thenoise to the texture of image during the forward process and recover maskedregions with unmasked ones of the texture via the reverse denoisingprocess.Despite the meaningful semantics generation,the existing arts sufferfrom the semantic discrepancy between masked and unmasked regions, since thesemantically dense unmasked texture fails to be completely degraded while themasked regions turn to the pure noise in diffusion process,leading to the largediscrepancy between them. In this paper,we aim to answer how unmasked semanticsguide texture denoising process;together with how to tackle the semanticdiscrepancy,to facilitate the consistent and meaningful semantics generation.To this end,we propose a novel structure-guided diffusion model namedStrDiffusion,to reformulate the conventional texture denoising process understructure guidance to derive a simplified denoising objective for imageinpainting,while revealing:1)the semantically sparse structure is beneficial totackle semantic discrepancy in early stage, while dense texture generatesreasonable semantics in late stage;2)the semantics from unmasked regionsessentially offer the time-dependent structure guidance for the texturedenoising process,benefiting from the time-dependent sparsity of the structuresemantics.For the denoising process,a structure-guided neural network istrained to estimate the simplified denoising objective by exploiting theconsistency of the denoised structure between masked and unmaskedregions.Besides,we devise an adaptive resampling strategy as a formal criterionas whether structure is competent to guide the texture denoising process,whileregulate their semantic correlations.Extensive experiments validate the meritsof StrDiffusion over the state-of-the-arts.Our code is available athttps://github.com/htyjers/StrDiffusion.