RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning

  • 2021-02-12 18:34:07
  • Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, Nando de Freitas
  • 0

Abstract

Offline methods for reinforcement learning have a potential to help bridgethe gap between reinforcement learning research and real-world applications.They make it possible to learn policies from offline datasets, thus overcomingconcerns associated with online data collection in the real-world, includingcost, safety, or ethical concerns. In this paper, we propose a benchmark calledRL Unplugged to evaluate and compare offline RL methods. RL Unplugged includesdata from a diverse range of domains including games ({\em e.g.,} Ataribenchmark) and simulated motor control problems ({\em e.g.,} DM Control Suite).The datasets include domains that are partially or fully observable, usecontinuous or discrete actions, and have stochastic vs. deterministic dynamics.We propose detailed evaluation protocols for each domain in RL Unplugged andprovide an extensive analysis of supervised learning and offline RL methodsusing these protocols. We will release data for all our tasks and open-sourceall algorithms presented in this paper. We hope that our suite of benchmarkswill increase the reproducibility of experiments and make it possible to studychallenging tasks with a limited computational budget, thus making RL researchboth more systematic and more accessible across the community. Moving forward,we view RL Unplugged as a living benchmark suite that will evolve and grow withdatasets contributed by the research community and ourselves. Our project pageis available on https://git.io/JJUhd.

 

Quick Read (beta)

loading the full paper ...