Abstract
Deep generative models have become increasingly effective at producingrealistic images from randomly sampled seeds, but using such models forcontrollable manipulation of existing images remains challenging. We proposethe Swapping Autoencoder, a deep model designed specifically for imagemanipulation, rather than random sampling. The key idea is to encode an imagewith two independent components and enforce that any swapped combination mapsto a realistic image. In particular, we encourage the components to representstructure and texture, by enforcing one component to encode co-occurrent patchstatistics across different parts of an image. As our method is trained with anencoder, finding the latent codes for a new input image becomes trivial, ratherthan cumbersome. As a result, it can be used to manipulate real input images invarious ways, including texture swapping, local and global editing, and latentcode vector arithmetic. Experiments on multiple datasets show that our modelproduces better results and is substantially more efficient compared to recentgenerative models.