No MCMC for me: Amortized sampling for fast and stable training of energy-based models

  • 2020-10-08 19:17:20
  • Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, David Duvenaud
  • 53

Abstract

Energy-Based Models (EBMs) present a flexible and appealing way torepresentuncertainty. Despite recent advances, training EBMs onhigh-dimensional dataremains a challenging problem as the state-of-the-artapproaches are costly, unstable, and require considerable tuning and domainexpertise to apply successfully. In this work we present a simple method fortraining EBMs at scale which uses an entropy-regularized generator to amortizethe MCMC sampling typically usedin EBM training. We improve upon priorMCMC-based entropy regularization methods with a fast variationalapproximation. We demonstrate the effectiveness of our approach by using it totrain tractable likelihood models. Next, we apply our estimator to the recentlyproposed Joint Energy Model (JEM), where we matchthe original performance withfaster and stable training. This allows us to extend JEM models tosemi-supervised classification on tabular data from a variety of continuousdomains.

 

Quick Read (beta)

loading the full paper ...