Inversion-Based Creativity Transfer with Diffusion Models

  • 2022-11-23 18:44:25
  • Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, Changsheng Xu
  • 63


In this paper, we introduce the task of "Creativity Transfer". The artisticcreativity within a painting is the means of expression, which includes notonly the painting material, colors, and brushstrokes, but also the high-levelattributes including semantic elements, object shape, etc. Previous arbitraryexample-guided artistic image generation methods (e.g., style transfer) oftenfail to control shape changes or convey semantic elements. The pre-trainedtext-to-image synthesis diffusion probabilistic models have achieved remarkablequality, but they often require extensive textual descriptions to accuratelyportray attributes of a particular painting. We believe that the uniqueness ofan artwork lies precisely in the fact that it cannot be adequately explainedwith normal language. Our key idea is to learn artistic creativity directlyfrom a single painting and then guide the synthesis without providing complextextual descriptions. Specifically, we assume creativity as a learnable textualdescription of a painting. We propose an attention-based inversion method,which can efficiently and accurately learn the holistic and detailedinformation of an image, thus capturing the complete artistic creativity of apainting. We demonstrate the quality and efficiency of our method on numerouspaintings of various artists and styles. Code and models are available at


Quick Read (beta)

loading the full paper ...