Abstract
Generalization in reinforcement learning (RL) is of importance for realdeployment of RL algorithms. Various schemes are proposed to address thegeneralization issues, including transfer learning, multi-task learning andmeta learning, as well as the robust and adversarial reinforcement learning.However, there is not a unified formulation of the various schemes, as well asthe comprehensive comparisons of methods across different schemes. In thiswork, we propose a game-theoretic framework for the generalization inreinforcement learning, named GiRL, where an RL agent is trained against anadversary over a set of tasks, where the adversary can manipulate thedistributions over tasks within a given threshold. With differentconfigurations, GiRL can reduce the various schemes mentioned above. To solveGiRL, we adapt the widely-used method in game theory, policy space responseoracle (PSRO) with the following three important modifications: i) we usemodel-agnostic meta learning (MAML) as the best-response oracle, ii) we proposea modified projected replicated dynamics, i.e., R-PRD, which ensures thecomputed meta-strategy of the adversary fall in the threshold, and iii) we alsopropose a protocol for the few-shot learning of the multiple strategies duringtesting. Extensive experiments on MuJoCo environments demonstrate that ourproposed methods can outperform existing baselines, e.g., MAML.