Abstract
Hierarchical reinforcement learning (HRL) is a promising approach to extendtraditional reinforcement learning (RL) methods to solve more complex tasks.Yet, the majority of current HRL methods require careful task-specific designand on-policy training, making them difficult to apply in real-world scenarios.In this paper, we study how we can develop HRL algorithms that are general, inthat they do not make onerous additional assumptions beyond standard RLalgorithms, and efficient, in the sense that they can be used with modestnumbers of interaction samples, making them suitable for real-world problemssuch as robotic control. For generality, we develop a scheme where lower-levelcontrollers are supervised with goals that are learned and proposedautomatically by the higher-level controllers. To address efficiency, wepropose to use off-policy experience for both higher and lower-level training.This poses a considerable challenge, since changes to the lower-level behaviorschange the action space for the higher-level policy, and we introduce anoff-policy correction to remedy this challenge. This allows us to takeadvantage of recent advances in off-policy model-free RL to learn both higher-and lower-level policies using substantially fewer environment interactionsthan on-policy algorithms. We term the resulting HRL agent HIRO and find thatit is generally applicable and highly sample-efficient. Our experiments showthat HIRO can be used to learn highly complex behaviors for simulated robots,such as pushing objects and utilizing them to reach target locations, learningfrom only a few million samples, equivalent to a few days of real-timeinteraction. In comparisons with a number of prior HRL methods, we find thatour approach substantially outperforms previous state-of-the-art techniques.