Successor Features for Transfer in Reinforcement Learning

  • 2018-04-12 11:41:05
  • André Barreto, Will Dabney, Rémi Munos, Jonathan J. Hunt, Tom Schaul, Hado van Hasselt, David Silver
  • 0

Abstract

Transfer in reinforcement learning refers to the notion that generalizationshould occur not only within a task but also across tasks. We propose atransfer framework for the scenario where the reward function changes betweentasks but the environment's dynamics remain the same. Our approach rests on twokey ideas: "successor features", a value function representation that decouplesthe dynamics of the environment from the rewards, and "generalized policyimprovement", a generalization of dynamic programming's policy improvementoperation that considers a set of policies rather than a single one. Puttogether, the two ideas lead to an approach that integrates seamlessly withinthe reinforcement learning framework and allows the free exchange ofinformation across tasks. The proposed method also provides performanceguarantees for the transferred policy even before any learning has taken place.We derive two theorems that set our approach in firm theoretical ground andpresent experiments that show that it successfully promotes transfer inpractice, significantly outperforming alternative methods in a sequence ofnavigation tasks and in the control of a simulated robotic arm.

 

Quick Read (beta)

loading the full paper ...