SinFusion: Training Diffusion Models on a Single Image or Video

  • 2022-11-21 18:59:33
  • Yaniv Nikankin, Niv Haim, Michal Irani
  • 44

Abstract

Diffusion models exhibited tremendous progress in image and video generation,exceeding GANs in quality and diversity. However, they are usually trained onvery large datasets and are not naturally adapted to manipulate a given inputimage or video. In this paper we show how this can be resolved by training adiffusion model on a single input image or video. Our image/video-specificdiffusion model (SinFusion) learns the appearance and dynamics of the singleimage or video, while utilizing the conditioning capabilities of diffusionmodels. It can solve a wide array of image/video-specific manipulation tasks.In particular, our model can learn from few frames the motion and dynamics of asingle input video. It can then generate diverse new video samples of the samedynamic scene, extrapolate short videos into long ones (both forward andbackward in time) and perform video upsampling. When trained on a single image,our model shows comparable performance and capabilities to previoussingle-image models in various image manipulation tasks.

 

Quick Read (beta)

loading the full paper ...