Epistemic Neural Networks

  • 2021-07-19 14:37:57
  • Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, Benjamin Van Roy
  • 45

Abstract

We introduce the \textit{epistemic neural network} (ENN) as an interface foruncertainty modeling in deep learning. All existing approaches to uncertaintymodeling can be expressed as ENNs, and any ENN can be identified with aBayesian neural network. However, this new perspective provides severalpromising directions for future research. Where prior work has developedprobabilistic inference tools for neural networks; we ask instead, `whichneural networks are suitable as tools for probabilistic inference?'. We proposea clear and simple metric for progress in ENNs: the KL-divergence with respectto a target distribution. We develop a computational testbed based on inferencein a neural network Gaussian process and release our code as a benchmark at\url{https://github.com/deepmind/enn}. We evaluate several canonical approachesto uncertainty modeling in deep learning, and find they vary greatly in theirperformance. We provide insight to the sensitivity of these results and showthat our metric is highly correlated with performance in sequential decisionproblems. Finally, we provide indications that new ENN architectures canimprove performance in both the statistical quality and computational cost.

 

Quick Read (beta)

loading the full paper ...