Investigating the Properties of Neural Network Representations in Reinforcement Learning

  • 2023-01-11 04:57:42
  • Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White
  • 0

Abstract

In this paper we investigate the properties of representations learned bydeep reinforcement learning systems. Much of the early work on representationsfor reinforcement learning focused on designing fixed-basis architectures toachieve properties thought to be desirable, such as orthogonality and sparsity.In contrast, the idea behind deep reinforcement learning methods is that theagent designer should not encode representational properties, but rather thatthe data stream should determine the properties of the representation -- goodrepresentations emerge under appropriate training schemes. In this paper webring these two perspectives together, empirically investigating the propertiesof representations that support transfer in reinforcement learning. Weintroduce and measure six representational properties over more than 25thousand agent-task settings. We consider Deep Q-learning agents with differentauxiliary losses in a pixel-based navigation environment, with source andtransfer tasks corresponding to different goal locations. We develop a methodto better understand why some representations work better for transfer, througha systematic approach varying task similarity and measuring and correlatingrepresentation properties with transfer performance. We demonstrate thegenerality of the methodology by investigating representations learned by aRainbow agent that successfully transfer across games modes in Atari 2600.

 

Quick Read (beta)

loading the full paper ...