Abstract
The ability to predict upcoming events has been hypothesized to comprise akey aspect of natural and machine cognition. This is supported by trends indeep reinforcement learning (RL), where self-supervised auxiliary objectivessuch as prediction are widely used to support representation learning andimprove task performance. Here, we study the effects predictive auxiliaryobjectives have on representation learning across different modules of an RLsystem and how these mimic representational changes observed in the brain. Wefind that predictive objectives improve and stabilize learning particularly inresource-limited architectures, and we identify settings where longerpredictive horizons better support representational transfer. Furthermore, wefind that representational changes in this RL system bear a strikingresemblance to changes in neural activity observed in the brain across variousexperiments. Specifically, we draw a connection between the auxiliarypredictive model of the RL system and hippocampus, an area thought to learn apredictive model to support memory-guided behavior. We also connect the encodernetwork and the value learning network of the RL system to visual cortex andstriatum in the brain, respectively. This work demonstrates how representationlearning in deep RL systems can provide an interpretable framework for modelingmulti-region interactions in the brain. The deep RL perspective taken here alsosuggests an additional role of the hippocampus in the brain -- that of anauxiliary learning system that benefits representation learning in otherregions.