Abstract
In this article, we aim to provide a literature review of differentformulations and approaches to continual reinforcement learning (RL), alsoknown as lifelong or non-stationary RL. We begin by discussing our perspectiveon why RL is a natural fit for studying continual learning. We then provide ataxonomy of different continual RL formulations by mathematicallycharacterizing two key properties of non-stationarity, namely, the scope anddriver non-stationarity. This offers a unified view of various formulations.Next, we review and present a taxonomy of continual RL approaches. We go on todiscuss evaluation of continual RL agents, providing an overview of benchmarksused in the literature and important metrics for understanding agentperformance. Finally, we highlight open problems and challenges in bridging thegap between the current state of continual RL and findings in neuroscience.While still in its early days, the study of continual RL has the promise todevelop better incremental reinforcement learners that can function inincreasingly realistic applications where non-stationarity plays a vital role.These include applications such as those in the fields of healthcare,education, logistics, and robotics.