Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

  • 2022-11-17 14:38:58
  • Souradeep Dutta, Kaustubh Sridhar, Osbert Bastani, Edgar Dobriban, James Weimer, Insup Lee, Julia Parish-Morris
  • 0

Abstract

Long horizon robot learning tasks with sparse rewards pose a significantchallenge for current reinforcement learning algorithms. A key feature enablinghumans to learn challenging control tasks is that they often receive expertintervention that enables them to understand the high-level structure of thetask before mastering low-level control actions. We propose a framework forleveraging expert intervention to solve long-horizon reinforcement learningtasks. We consider \emph{option templates}, which are specifications encoding apotential option that can be trained using reinforcement learning. We formulateexpert intervention as allowing the agent to execute option templates beforelearning an implementation. This enables them to use an option, beforecommitting costly resources to learning it. We evaluate our approach on threechallenging reinforcement learning problems, showing that it outperformsstate-of-the-art approaches by two orders of magnitude. Videos of trainedagents and our code can be found at:https://sites.google.com/view/stickymittens

 

Quick Read (beta)

loading the full paper ...