Autonomous Reinforcement Learning: Formalism and Benchmarking

  • 2022-08-08 01:26:16
  • Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, Chelsea Finn
  • 0

Abstract

Reinforcement learning (RL) provides a naturalistic framing for learningthrough trial and error, which is appealing both because of its simplicity andeffectiveness and because of its resemblance to how humans and animals acquireskills through experience. However, real-world embodied learning, such as thatperformed by humans and animals, is situated in a continual, non-episodicworld, whereas common benchmark tasks in RL are episodic, with the environmentresetting between trials to provide the agent with multiple attempts. Thisdiscrepancy presents a major challenge when attempting to take RL algorithmsdeveloped for episodic simulated environments and run them on real-worldplatforms, such as robots. In this paper, we aim to address this discrepancy bylaying out a framework for Autonomous Reinforcement Learning (ARL):reinforcement learning where the agent not only learns through its ownexperience, but also contends with lack of human supervision to reset betweentrials. We introduce a simulated benchmark EARL around this framework,containing a set of diverse and challenging simulated tasks reflective of thehurdles introduced to learning when only a minimal reliance on extrinsicintervention can be assumed. We show that standard approaches to episodic RLand existing approaches struggle as interventions are minimized, underscoringthe need for developing new algorithms for reinforcement learning with agreater focus on autonomy.

 

Quick Read (beta)

loading the full paper ...