Abstract
Robust Reinforcement Learning (RL) focuses on improving performances undermodel errors or adversarial attacks, which facilitates the real-life deploymentof RL agents. Robust Adversarial Reinforcement Learning (RARL) is one of themost popular frameworks for robust RL. However, most of the existing literaturemodels RARL as a zero-sum simultaneous game with Nash equilibrium as thesolution concept, which could overlook the sequential nature of RL deployments,produce overly conservative agents, and induce training instability. In thispaper, we introduce a novel hierarchical formulation of robust RL - ageneral-sum Stackelberg game model called RRL-Stack - to formalize thesequential nature and provide extra flexibility for robust training. We developthe Stackelberg Policy Gradient algorithm to solve RRL-Stack, leveraging theStackelberg learning dynamics by considering the adversary's response. Ourmethod generates challenging yet solvable adversarial environments whichbenefit RL agents' robust learning. Our algorithm demonstrates better trainingstability and robustness against different testing conditions in thesingle-agent robotics control and multi-agent highway merging tasks.