Abstract
Reinforcement learning has a rich history in neuroscience, from early work ondopamine as a reward prediction error signal for temporal difference learning(Schultz et al., 1997) to recent work suggesting that dopamine could implementa form of 'distributional reinforcement learning' popularized in deep learning(Dabney et al., 2020). Throughout this literature, there has been a tight linkbetween theoretical advances in reinforcement learning and neuroscientificexperiments and findings. As a result, the theories describing our experimentaldata have become increasingly complex and difficult to navigate. In thisreview, we cover the basic theory underlying classical work in reinforcementlearning and build up to an introductory overview of methods used in moderndeep reinforcement learning that have found applications in systemsneuroscience. We start with an overview of the reinforcement learning problemand classical temporal difference algorithms, followed by a discussion of'model-free' and 'model-based' reinforcement learning together with methodssuch as DYNA and successor representations that fall in between these twocategories. Throughout these sections, we highlight the close parallels betweenthe machine learning methods and related work in both experimental andtheoretical neuroscience. We then provide an introduction to deep reinforcementlearning with examples of how these methods have been used to model differentlearning phenomena in the systems neuroscience literature, such asmeta-reinforcement learning (Wang et al., 2018) and distributionalreinforcement learning (Dabney et al., 2020). Code that implements the methodsdiscussed in this work and generates the figures is also provided.