Adversarially Guided Subgoal Generation for Hierarchical Reinforcement Learning

  • 2022-01-24 12:30:38
  • Vivienne Huiling Wang, Joni Pajarinen, Tinghuai Wang, Joni Kämäräinen
  • 1

Abstract

Hierarchical reinforcement learning (HRL) proposes to solve difficult tasksby performing decision-making and control at successively higher levels oftemporal abstraction. However, off-policy training in HRL often suffers fromthe problem of non-stationary high-level decision making since the low-levelpolicy is constantly changing. In this paper, we propose a novel HRL approachfor mitigating the non-stationarity by adversarially enforcing the high-levelpolicy to generate subgoals compatible with the current instantiation of thelow-level policy. In practice, the adversarial learning can be implemented bytraining a simple discriminator network concurrently with the high-level policywhich determines the compatibility level of subgoals. Experiments withstate-of-the-art algorithms show that our approach significantly improveslearning efficiency and overall performance of HRL in various challengingcontinuous control tasks.

 

Quick Read (beta)

loading the full paper ...