Abstract
Directed latent variable models that formulate the joint distribution as$p(x,z) = p(z) p(x \mid z)$ have the advantage of fast and exact sampling.However, these models have the weakness of needing to specify $p(z)$, oftenwith a simple fixed prior that limits the expressiveness of the model.Undirected latent variable models discard the requirement that $p(z)$ bespecified with a prior, yet sampling from them generally requires an iterativeprocedure such as blocked Gibbs-sampling that may require many steps to drawsamples from the joint distribution $p(x, z)$. We propose a novel approach tolearning the joint distribution between the data and a latent code which usesan adversarially learned iterative procedure to gradually refine the jointdistribution, $p(x, z)$, to better match with the data distribution on eachstep. GibbsNet is the best of both worlds both in theory and in practice.Achieving the speed and simplicity of a directed latent variable model, it isguaranteed (assuming the adversarial game reaches the virtual training criteriaglobal minimum) to produce samples from $p(x, z)$ with only a few samplingiterations. Achieving the expressiveness and flexibility of an undirectedlatent variable model, GibbsNet does away with the need for an explicit $p(z)$and has the ability to do attribute prediction, class-conditional generation,and joint image-attribute modeling in a single model which is not trained forany of these specific tasks. We show empirically that GibbsNet is able to learna more complex $p(z)$ and show that this leads to improved inpainting anditerative refinement of $p(x, z)$ for dozens of steps and stable generationwithout collapse for thousands of steps, despite being trained on only a fewsteps.