Paint by Example: Exemplar-based Image Editing with Diffusion Models

  • 2022-11-23 18:59:52
  • Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, Fang Wen
  • 50

Abstract

Language-guided image editing has achieved great success recently. In thispaper, for the first time, we investigate exemplar-guided image editing formore precise control. We achieve this goal by leveraging self-supervisedtraining to disentangle and re-organize the source image and the exemplar.However, the naive approach will cause obvious fusing artifacts. We carefullyanalyze it and propose an information bottleneck and strong augmentations toavoid the trivial solution of directly copying and pasting the exemplar image.Meanwhile, to ensure the controllability of the editing process, we design anarbitrary shape mask for the exemplar image and leverage the classifier-freeguidance to increase the similarity to the exemplar image. The whole frameworkinvolves a single forward of the diffusion model without any iterativeoptimization. We demonstrate that our method achieves an impressive performanceand enables controllable editing on in-the-wild images with high fidelity.

 

Quick Read (beta)

loading the full paper ...