Abstract
Demonstration-guided reinforcement learning (RL) is a promising approach forlearning complex behaviors by leveraging both reward feedback and a set oftarget task demonstrations. Prior approaches for demonstration-guided RL treatevery new task as an independent learning problem and attempt to follow theprovided demonstrations step-by-step, akin to a human trying to imitate acompletely unseen behavior by following the demonstrator's exact musclemovements. Naturally, such learning will be slow, but often new behaviors arenot completely unseen: they share subtasks with behaviors we have previouslylearned. In this work, we aim to exploit this shared subtask structure toincrease the efficiency of demonstration-guided RL. We first learn a set ofreusable skills from large offline datasets of prior experience collectedacross many tasks. We then propose Skill-based Learning with Demonstrations(SkiLD), an algorithm for demonstration-guided RL that efficiently leveragesthe provided demonstrations by following the demonstrated skills instead of theprimitive actions, resulting in substantial performance improvements over priordemonstration-guided RL approaches. We validate the effectiveness of ourapproach on long-horizon maze navigation and complex robot manipulation tasks.