Abstract
We address the problem of learning hierarchical deep neural network policiesfor reinforcement learning. Our aim is to design a hierarchical reinforcementlearning algorithm that can construct hierarchical representations in bottom-uplayerwise fashion. In contrast to methods that explicitly restrict or cripplelower layers of a hierarchy to force them to use higher-level modulatingsignals, each layer in our framework is trained to directly solve the task, butacquires a range of diverse strategies via a maximum entropy reinforcementlearning objective. Each layer is also augmented with latent random variables,which are sampled from a prior distribution during the training of that layer.The maximum entropy objective causes these latent variables to be incorporatedinto the layer's policy, and the higher level layer can directly control thebehavior of the lower layer through this latent space. Furthermore, byconstraining the mapping from latent variables to actions to be invertible,higher layers retain full expressivity: neither the higher layers nor the lowerlayers are constrained in their behavior. Our experimental evaluationdemonstrates that we can improve on the performance of single-layer policies onstandard benchmark tasks simply by adding additional layers, and that ourmethod can solve more complex sparse-reward tasks by learning higher-levelpolicies on top of high-entropy skills optimized for simple low-levelobjectives.