Data-Efficient Reinforcement Learning with Momentum Predictive Representations

  • 2020-07-23 03:09:35
  • Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, Philip Bachman
  • 0

Abstract

While deep reinforcement learning excels at solving tasks where large amountsof data can be collected through virtually unlimited interaction with theenvironment, learning from limited interaction remains a key challenge. Weposit that an agent can learn more efficiently if we augment rewardmaximization with self-supervised objectives based on structure in its visualinput and sequential interaction with the environment. Our method, MomentumPredictive Representations (MPR), trains an agent to predict its own latentstate representations multiple steps into the future. We compute targetrepresentations for future states using an encoder which is an exponentialmoving average of the agent's parameters, and we make predictions using alearned transition model. On its own, this future prediction objectiveoutperforms prior methods for sample-efficient deep RL from pixels. We furtherimprove performance by adding data augmentation to the future prediction loss,which forces the agent's representations to be consistent across multiple viewsof an observation. Our full self-supervised objective, which combines futureprediction and data augmentation, achieves a median human-normalized score of0.444 on Atari in a setting limited to 100K steps of environment interaction,which is a 66% relative improvement over the previous state-of-the-art.Moreover, even in this limited data regime, MPR exceeds expert human scores on6 out of 26 games.

 

Quick Read (beta)

loading the full paper ...