Abstract
In a standard view of the reinforcement learning problem, an agent's goal isto efficiently identify a policy that maximizes long-term reward. However, thisperspective is based on a restricted view of learning as finding a solution,rather than treating learning as endless adaptation. In contrast, continualreinforcement learning refers to the setting in which the best agents neverstop learning. Despite the importance of continual reinforcement learning, thecommunity lacks a simple definition of the problem that highlights itscommitments and makes its primary concepts precise and clear. To this end, thispaper is dedicated to carefully defining the continual reinforcement learningproblem. We formalize the notion of agents that "never stop learning" through anew mathematical language for analyzing and cataloging agents. Using this newlanguage, we define a continual learning agent as one that can be understood ascarrying out an implicit search process indefinitely, and continualreinforcement learning as the setting in which the best agents are allcontinual learning agents. We provide two motivating examples, illustratingthat traditional views of multi-task reinforcement learning and continualsupervised learning are special cases of our definition. Collectively, thesedefinitions and perspectives formalize many intuitive concepts at the heart oflearning, and open new research pathways surrounding continual learning agents.