Offline Meta-Reinforcement Learning with Online Self-Supervision

  • 2021-07-19 21:42:36
  • Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang, Sergey Levine
  • 0

Abstract

Meta-reinforcement learning (RL) can meta-train policies that adapt to newtasks with orders of magnitude less data than standard RL, but meta-trainingitself is costly and time-consuming. If we can meta-train on offline data, thenwe can reuse the same static dataset, labeled once with rewards for differenttasks, to meta-train policies that adapt to a variety of new tasks at meta-testtime. Although this capability would make meta-RL a practical tool forreal-world use, offline meta-RL presents additional challenges beyond onlinemeta-RL or standard offline RL settings. Meta-RL learns an exploration strategythat collects data for adapting, and also meta-trains a policy that quicklyadapts to data from a new task. Since this policy was meta-trained on a fixed,offline dataset, it might behave unpredictably when adapting to data collectedby the learned exploration strategy, which differs systematically from theoffline data and thus induces distributional shift. We do not want to removethis distributional shift by simply adopting a conservative explorationstrategy, because learning an exploration strategy enables an agent to collectbetter data for faster adaptation. Instead, we propose a hybrid offline meta-RLalgorithm, which uses offline data with rewards to meta-train an adaptivepolicy, and then collects additional unsupervised online data, without anyreward labels to bridge this distribution shift. By not requiring reward labelsfor online collection, this data can be much cheaper to collect. We compare ourmethod to prior work on offline meta-RL on simulated robot locomotion andmanipulation tasks and find that using additional unsupervised online datacollection leads to a dramatic improvement in the adaptive capabilities of themeta-trained policies, matching the performance of fully online meta-RL on arange of challenging domains that require generalization to new tasks.

 

Quick Read (beta)

loading the full paper ...