Boosting Black Box Variational Inference

  • 2018-06-12 17:01:20
  • Francesco Locatello, Gideon Dresdner, Rajiv Khanna, Isabel Valera, Gunnar R├Ątsch
  • 0


Approximating a probability density in a tractable manner is a central taskin Bayesian statistics. Variational Inference (VI) is a popular technique thatachieves tractability by choosing a relatively simple variational family.Borrowing ideas from the classic boosting framework, recent approaches attemptto \emph{boost} VI by replacing the selection of a single density with agreedily constructed mixture of densities. In order to guarantee convergence,previous works impose stringent assumptions that require significant effort forpractitioners. Specifically, they require a custom implementation of the greedystep (called the LMO) for every probabilistic model with respect to anunnatural variational family of truncated distributions. Our work fixes theseissues with novel theoretical and algorithmic insights. On the theoreticalside, we show that boosting VI satisfies a relaxed smoothness assumption whichis sufficient for the convergence of the functional Frank-Wolfe (FW) algorithm.Furthermore, we rephrase the LMO problem and propose to maximize the ResidualELBO (RELBO) which replaces the standard ELBO optimization in VI. Thesetheoretical enhancements allow for black box implementation of the boostingsubroutine. Finally, we present a stopping criterion drawn from the duality gapin the classic FW analyses and exhaustive experiments to illustrate theusefulness of our theoretical and algorithmic contributions.


Introduction (beta)



Conclusion (beta)