Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

  • 2018-08-02 08:21:25
  • Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, Akash Srivastava
  • 0

Abstract

Uncertainty computation in deep learning is essential to design robust andreliable systems. Variational inference (VI) is a promising approach for suchcomputation, but requires more effort to implement and execute compared tomaximum-likelihood methods. In this paper, we propose new natural-gradientalgorithms to reduce such efforts for Gaussian mean-field VI. Our algorithmscan be implemented within the Adam optimizer by perturbing the network weightsduring gradient evaluations, and uncertainty estimates can be cheaply obtainedby using the vector that adapts the learning rate. This requires lower memory,computation, and implementation effort than existing VI methods, whileobtaining uncertainty estimates of comparable quality. Our empirical resultsconfirm this and further suggest that the weight-perturbation in our algorithmcould be useful for exploration in reinforcement learning and stochasticoptimization.

 

Quick Read (beta)

loading the full paper ...