Abstract
Latent space Energy-Based Models (EBMs), also known as energy-based priors,have drawn growing interests in generative modeling. Fueled by its flexibilityin the formulation and strong modeling power of the latent space, recent worksbuilt upon it have made interesting attempts aiming at the interpretability oftext modeling. However, latent space EBMs also inherit some flaws from EBMs indata space; the degenerate MCMC sampling quality in practice can lead to poorgeneration quality and instability in training, especially on data with complexlatent structures. Inspired by the recent efforts that leverage diffusionrecovery likelihood learning as a cure for the sampling issue, we introduce anovel symbiosis between the diffusion models and latent space EBMs in avariational learning framework, coined as the latent diffusion energy-basedmodel. We develop a geometric clustering-based regularization jointly with theinformation bottleneck to further improve the quality of the learned latentspace. Experiments on several challenging tasks demonstrate the superiorperformance of our model on interpretable text modeling over strongcounterparts.