Count-Based Exploration with the Successor Representation

  • 2018-08-14 02:56:53
  • Marlos C. Machado, Marc G. Bellemare, Michael Bowling
  • 0

Abstract

The problem of exploration in reinforcement learning is well-understood inthe tabular case and many sample-efficient algorithms are known. Nevertheless,it is often unclear how the algorithms in the tabular setting can be extendedto tasks with large state-spaces where generalization is required. Recentpromising developments generally depend on problem-specific density models orhandcrafted features. In this paper we introduce a simple approach forexploration that allows us to develop theoretically justified algorithms in thetabular case but that also give us intuitions for new algorithms applicable tosettings where function approximation is required. Our approach and itsunderlying theory is based on the substochastic successor representation, aconcept we develop here. While the traditional successor representation is arepresentation that defines state generalization by the similarity of successorstates, the substochastic successor representation is also able to implicitlycount the number of times each state (or feature) has been observed. Thisextension connects two until now disjoint areas of research. We show intraditional tabular domains (RiverSwim and SixArms) that our algorithmempirically performs as well as other sample-efficient algorithms. We thendescribe a deep reinforcement learning algorithm inspired by these ideas andshow that it matches the performance of recent pseudo-count-based methods inhard exploration Atari 2600 games.

 

Quick Read (beta)

loading the full paper ...