Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

  • 2022-08-06 20:45:26
  • Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, Yee Whye Teh
  • 0

Abstract

Offline reinforcement learning has shown great promise in leveraging largepre-collected datasets for policy learning, allowing agents to forgooften-expensive online data collection. However, to date, offline reinforcementlearning from visual observations with continuous action spaces has beenrelatively under-explored, and there is a lack of understanding of where theremaining challenges lie. In this paper, we seek to establish simple baselinesfor continuous control in the visual domain. We show that simple modificationsto two state-of-the-art vision-based online reinforcement learning algorithms,DreamerV2 and DrQ-v2, suffice to outperform prior work and establish acompetitive baseline. We rigorously evaluate these algorithms on both existingoffline datasets and a new testbed for offline reinforcement learning fromvisual observations that better represents the data distributions present inreal-world offline RL problems, and open-source our code and data to facilitateprogress in this important domain. Finally, we present and analyze several keydesiderata unique to offline RL from visual observations, including visualdistractions and visually identifiable changes in dynamics.

 

Quick Read (beta)

loading the full paper ...