Abstract
Uncertainty computation in deep learning is essential to design robust andreliable systems. Variational inference (VI) is a promising approach for suchcomputation, but requires more effort to implement and execute compared tomaximum-likelihood methods. In this paper, we propose new natural-gradientalgorithms to reduce such efforts for Gaussian mean-field VI. Our algorithmscan be implemented within the Adam optimizer by perturbing the network weightsduring gradient evaluations, and uncertainty estimates can be cheaply obtainedby using the vector that adapts the learning rate. This requires lower memory,computation, and implementation effort than existing VI methods, whileobtaining uncertainty estimates of comparable quality. Our empirical resultsconfirm this and further suggest that the weight-perturbation in our algorithmcould be useful for exploration in reinforcement learning and stochasticoptimization.