Abstract
Understanding the uncertainty of a neural network's (NN) predictions isessential for many purposes. The Bayesian framework provides a principledapproach to this, however applying it to NNs is challenging due to largenumbers of parameters and data. Ensembling NNs provides an easilyimplementable, scalable method for uncertainty quantification, however, it hasbeen criticised for not being Bayesian. This work proposes one modification tothe usual process that we argue does result in approximate Bayesian inference;regularising parameters about values drawn from a distribution which can be setequal to the prior. A theoretical analysis of the procedure in a simplifiedsetting suggests the recovered posterior is centred correctly but tends to havean underestimated marginal variance, and overestimated correlation. However,two conditions can lead to exact recovery. We argue that these conditions arepartially present in NNs. Empirical evaluations demonstrate it has an advantageover standard ensembling, and is competitive with variational methods.