Offline Meta-Reinforcement Learning with Advantage Weighting

  • 2020-10-06 17:35:24
  • Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, Chelsea Finn
  • 0

Abstract

This paper introduces the offline meta-reinforcement learning (offlinemeta-RL) problem setting and proposes an algorithm that performs well in thissetting. Offline meta-RL is analogous to the widely successful supervisedlearning strategy of pre-training a model on a large batch of fixed,pre-collected data (possibly from various tasks) and fine-tuning the model to anew task with relatively little data. That is, in offline meta-RL, wemeta-train on fixed, pre-collected data from several tasks and adapt to a newtask with a very small amount (less than 5 trajectories) of data from the newtask. By nature of being offline, algorithms for offline meta-RL can utilizethe largest possible pool of training data available and eliminate potentiallyunsafe or costly data collection during meta-training. This setting inheritsthe challenges of offline RL, but it differs significantly because offline RLdoes not generally consider a) transfer to new tasks or b) limited data fromthe test task, both of which we face in offline meta-RL. Targeting the offlinemeta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW).MACAW is an optimization-based meta-learning algorithm that uses simple,supervised regression objectives for both the inner and outer loop ofmeta-training. On offline variants of common meta-RL benchmarks, we empiricallyfind that this approach enables fully offline meta-reinforcement learning andachieves notable gains over prior methods.

 

Quick Read (beta)

loading the full paper ...