Human-Timescale Adaptation in an Open-Ended Task Space

  • 2023-01-18 15:39:21
  • Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rockt√§schel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, Lei Zhang
  • 30

Abstract

Foundation models have shown impressive adaptation and scalability insupervised and self-supervised learning problems, but so far these successeshave not fully translated to reinforcement learning (RL). In this work, wedemonstrate that training an RL agent at scale leads to a general in-contextlearning algorithm that can adapt to open-ended novel embodied 3D problems asquickly as humans. In a vast space of held-out environment dynamics, ouradaptive agent (AdA) displays on-the-fly hypothesis-driven exploration,efficient exploitation of acquired knowledge, and can successfully be promptedwith first-person demonstrations. Adaptation emerges from three ingredients:(1) meta-reinforcement learning across a vast, smooth and diverse taskdistribution, (2) a policy parameterised as a large-scale attention-basedmemory architecture, and (3) an effective automated curriculum that prioritisestasks at the frontier of an agent's capabilities. We demonstrate characteristicscaling laws with respect to network size, memory length, and richness of thetraining task distribution. We believe our results lay the foundation forincreasingly general and adaptive RL agents that perform well acrossever-larger open-ended domains.

 

Quick Read (beta)

loading the full paper ...