Abstract
Posterior collapse in Variational Autoencoders (VAEs) arises when thevariational posterior distribution closely matches the prior for a subset oflatent variables. This paper presents a simple and intuitive explanation forposterior collapse through the analysis of linear VAEs and their directcorrespondence with Probabilistic PCA (pPCA). We explain how posterior collapsemay occur in pPCA due to local maxima in the log marginal likelihood.Unexpectedly, we prove that the ELBO objective for the linear VAE does notintroduce additional spurious local maxima relative to log marginal likelihood.We show further that training a linear VAE with exact variational inferencerecovers an identifiable global maximum corresponding to the principalcomponent directions. Empirically, we find that our linear analysis ispredictive even for high-capacity, non-linear VAEs and helps explain therelationship between the observation noise, local maxima, and posteriorcollapse in deep Gaussian VAEs.