Continual Model-Based Reinforcement Learning with Hypernetworks

  • 2021-03-30 02:46:27
  • Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, Florian Shkurti
  • 0


Effective planning in model-based reinforcement learning (MBRL) andmodel-predictive control (MPC) relies on the accuracy of the learned dynamicsmodel. In many instances of MBRL and MPC, this model is assumed to bestationary and is periodically re-trained from scratch on state transitionexperience collected from the beginning of environment interactions. Thisimplies that the time required to train the dynamics model - and the pauserequired between plan executions - grows linearly with the size of thecollected experience. We argue that this is too slow for lifelong robotlearning and propose HyperCRL, a method that continually learns the encountereddynamics in a sequence of tasks using task-conditional hypernetworks. Ourmethod has three main attributes: first, it includes dynamics learning sessionsthat do not revisit training data from previous tasks, so it only needs tostore the most recent fixed-size portion of the state transition experience;second, it uses fixed-capacity hypernetworks to represent non-stationary andtask-aware dynamics; third, it outperforms existing continual learningalternatives that rely on fixed-capacity networks, and does competitively withbaselines that remember an ever increasing coreset of past experience. We showthat HyperCRL is effective in continual model-based reinforcement learning inrobot locomotion and manipulation scenarios, such as tasks involving pushingand door opening. Our project website with videos is at this link


Quick Read (beta)

loading the full paper ...