Abstract
This paper demonstrates a novel method for separately estimating aleatoricrisk and epistemic uncertainty in deep reinforcement learning. Aleatoric risk,which arises from inherently stochastic environments or agents, must beaccounted for in the design of risk-sensitive algorithms. Epistemicuncertainty, which stems from limited data, is important both forrisk-sensitivity and to efficiently explore an environment. We first present aBayesian framework for learning the return distribution in reinforcementlearning, which provides theoretical foundations for quantifying both types ofuncertainty. Based on this framework, we show that the disagreement betweenonly two neural networks is sufficient to produce a low-variance estimate ofthe epistemic uncertainty on the return distribution, thus providing a simpleand computationally cheap uncertainty metric. We demonstrate experiments thatillustrate our method and some applications.