Tighter Variational Bounds are Not Necessarily Better

  • 2018-02-13 10:17:32
  • Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, Yee Whye Teh
  • 49

Abstract

We provide theoretical and empirical evidence that using tighter evidencelower bounds (ELBOs) can be detrimental to the process of learning an inferencenetwork by reducing the signal-to-noise ratio of the gradient estimator. Ourresults call into question common implicit assumptions that tighter ELBOs arebetter variational objectives for simultaneous model learning and inferenceamortization schemes. Based on our insights, we introduce three new algorithms:the partially importance weighted auto-encoder (PIWAE), the multiply importanceweighted auto-encoder (MIWAE), and the combination importance weightedauto-encoder (CIWAE), each of which includes the standard importance weightedauto-encoder (IWAE) as a special case. We show that each can deliverimprovements over IWAE, even when performance is measured by the IWAE targetitself. Moreover, PIWAE can simultaneously deliver improvements in both thequality of the inference network and generative network, relative to IWAE.

 

Quick Read (beta)

loading the full paper ...