Sub-policy Adaptation for Hierarchical Reinforcement Learning

  • 2020-05-13 23:32:41
  • Alexander C. Li, Carlos Florensa, Ignasi Clavera, Pieter Abbeel
  • 0


Hierarchical reinforcement learning is a promising approach to tacklelong-horizon decision-making problems with sparse rewards. Unfortunately, mostmethods still decouple the lower-level skill acquisition process and thetraining of a higher level that controls the skills in a new task. Leaving theskills fixed can lead to significant sub-optimality in the transfer setting. Inthis work, we propose a novel algorithm to discover a set of skills, andcontinuously adapt them along with the higher level even when training on a newtask. Our main contributions are two-fold. First, we derive a new hierarchicalpolicy gradient with an unbiased latent-dependent baseline, and we introduceHierarchical Proximal Policy Optimization (HiPPO), an on-policy method toefficiently train all levels of the hierarchy jointly. Second, we propose amethod for training time-abstractions that improves the robustness of theobtained skills to environment changes. Code and results are available


Quick Read (beta)

loading the full paper ...