Abstract
Hierarchical reinforcement learning (HRL) helps address large-scale andsparse reward issues in reinforcement learning. In HRL, the policy model has aninner representation structured in levels. With this structure, thereinforcement learning task is expected to be decomposed into correspondinglevels with sub-tasks, and thus the learning can be more efficient. In HRL,although it is intuitive that a high-level policy only needs to make macrodecisions in a low frequency, the exact frequency is hard to be simplydetermined. Previous HRL approaches often employed a fixed-time skip strategyor learn a terminal condition without taking account of the context, which,however, not only requires manual adjustments but also sacrifices some decisiongranularity. In this paper, we propose the \emph{temporal-adaptive hierarchicalpolicy learning} (TEMPLE) structure, which uses a temporal gate to adaptivelycontrol the high-level policy decision frequency. We train the TEMPLE structurewith PPO and test its performance in a range of environments including 2-Drooms, Mujoco tasks, and Atari games. The results show that the TEMPLEstructure can lead to improved performance in these environments with asequential adaptive high-level control.