Abstract
Effective planning in model-based reinforcement learning (MBRL) andmodel-predictive control (MPC) relies on the accuracy of the learned dynamicsmodel. In many instances of MBRL and MPC, this model is assumed to bestationary and is periodically re-trained from scratch on state transitionexperience collected from the beginning of environment interactions. Thisimplies that the time required to train the dynamics model - and the pauserequired between plan executions - grows linearly with the size of thecollected experience. We argue that this is too slow for lifelong robotlearning and propose HyperCRL, a method that continually learns the encountereddynamics in a sequence of tasks using task-conditional hypernetworks. Ourmethod has three main attributes: first, it enables constant-time dynamicslearning sessions between planning and only needs to store the most recentfixed-size portion of the state transition experience; second, it usesfixed-capacity hypernetworks to represent non-stationary and task-awaredynamics; third, it outperforms existing continual learning alternatives thatrely on fixed-capacity networks, and does competitively with baselines thatremember an ever increasing coreset of past experience. We show that HyperCRLis effective in continual model-based reinforcement learning in robotlocomotion and manipulation scenarios, such as tasks involving pushing and dooropening. Our project website with code and videos is at this linkhttp://rvl.cs.toronto.edu/blog/2020/hypercrl/