Abstract
A deep generative model such as a GAN learns to model a rich set of semanticand physical rules about the target distribution, but up to now, it has beenobscure how such rules are encoded in the network, or how a rule could bechanged. In this paper, we introduce a new problem setting: manipulation ofspecific rules encoded by a deep generative model. To address the problem, wepropose a formulation in which the desired rule is changed by manipulating alayer of a deep network as a linear associative memory. We derive an algorithmfor modifying one entry of the associative memory, and we demonstrate thatseveral interesting structural rules can be located and modified within thelayers of state-of-the-art generative models. We present a user interface toenable users to interactively change the rules of a generative model to achievedesired effects, and we show several proof-of-concept applications. Finally,results on multiple datasets demonstrate the advantage of our method againststandard fine-tuning methods and edit transfer algorithms.