Abstract
The generalized Gauss-Newton (GGN) approximation is often used to makepractical Bayesian deep learning approaches scalable by replacing a secondorder derivative with a product of first order derivatives. In this paper weargue that the GGN approximation should be understood as a local linearizationof the underlying Bayesian neural network (BNN), which turns the BNN into ageneralized linear model (GLM). Because we use this linearized model forposterior inference, we should also predict using this modified model insteadof the original one. We refer to this modified predictive as "GLM predictive"and show that it effectively resolves common underfitting problems of theLaplace approximation. It extends previous results in this vein to generallikelihoods and has an equivalent Gaussian process formulation, which enablesalternative inference schemes for BNNs in function space. We demonstrate theeffectiveness of our approach on several standard classification datasets aswell as on out-of-distribution detection. We provide an implementation athttps://github.com/AlexImmer/BNN-predictions.