Abstract
Variational Bayesian neural networks (BNNs) perform variational inferenceover weights, but it is difficult to specify meaningful priors and approximateposteriors in a high-dimensional weight space. We introduce functionalvariational Bayesian neural networks (fBNNs), which maximize an Evidence LowerBOund (ELBO) defined directly on stochastic processes, i.e. distributions overfunctions. We prove that the KL divergence between stochastic processes equalsthe supremum of marginal KL divergences over all finite sets of inputs. Basedon this, we introduce a practical training objective which approximates thefunctional ELBO using finite measurement sets and the spectral Stein gradientestimator. With fBNNs, we can specify priors entailing rich structures,including Gaussian processes and implicit stochastic processes. Empirically, wefind fBNNs extrapolate well using various structured priors, provide reliableuncertainty estimates, and scale to large datasets.