Abstract
To improve policy robustness of deep reinforcement learning agents, a line ofrecent works focus on producing disturbances of the environment. Existingapproaches of the literature to generate meaningful disturbances of theenvironment are adversarial reinforcement learning methods. These methods setthe problem as a two-player game between the protagonist agent, which learns toperform a task in an environment, and the adversary agent, which learns todisturb the protagonist via modifications of the considered environment. Bothprotagonist and adversary are trained with deep reinforcement learningalgorithms. Alternatively, we propose in this paper to build on gradient-basedadversarial attacks, usually used for classification tasks for instance, thatwe apply on the critic network of the protagonist to identify efficientdisturbances of the environment. Rather than learning an attacker policy, whichusually reveals as very complex and unstable, we leverage the knowledge of thecritic network of the protagonist, to dynamically complexify the task at eachstep of the learning process. We show that our method, while being faster andlighter, leads to significantly better improvements in policy robustness thanexisting methods of the literature.