Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations

  • 2024-10-14 18:56:24
  • Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, Wen-Sheng Chu
  • 0

Abstract

Generative models transform random noise into images; their inversion aims totransform images back to structured noise for recovery and editing. This paperaddresses two key tasks: (i) inversion and (ii) editing of a real image usingstochastic equivalents of rectified flow models (such as Flux). AlthoughDiffusion Models (DMs) have recently dominated the field of generative modelingfor images, their inversion presents faithfulness and editability challengesdue to nonlinearities in drift and diffusion. Existing state-of-the-art DMinversion approaches rely on training of additional parameters or test-timeoptimization of latent variables; both are expensive in practice. RectifiedFlows (RFs) offer a promising alternative to diffusion models, yet theirinversion has been underexplored. We propose RF inversion using dynamic optimalcontrol derived via a linear quadratic regulator. We prove that the resultingvector field is equivalent to a rectified stochastic differential equation.Additionally, we extend our framework to design a stochastic sampler for Flux.Our inversion method allows for state-of-the-art performance in zero-shotinversion and editing, outperforming prior works in stroke-to-image synthesisand semantic image editing, with large-scale human evaluations confirming userpreference.

 

Quick Read (beta)

loading the full paper ...