Addressing Sample Inefficiency and Reward Bias in Inverse Reinforcement Learning

  • 2018-09-09 05:37:25
  • Ilya Kostrikov, Kumar Krishna Agrawal, Sergey Levine, Jonathan Tompson
  • 0

Abstract

The Generative Adversarial Imitation Learning (GAIL) framework from Ho &Ermon (2016) is known for being surprisingly sample efficient in terms ofdemonstrations provided by an expert policy. However, the algorithm requires asignificantly larger number of policy interactions with the environment inorder to imitate the expert. In this work we address this problem by proposinga sample efficient algorithm for inverse reinforcement learning thatincorporates both off-policy reinforcement learning and adversarial imitationlearning. We also show that GAIL has a number of biases associated with thechoice of reward function, which can unintentionally encode prior knowledge ofsome tasks, and prevent learning in others. We address these shortcomings byanalyzing the issue and correcting invalid assumptions used when defining thelearned reward function. We demonstrate that our algorithm achievesstate-of-the-art performance for an inverse reinforcement learning framework ona variety of standard benchmark tasks, and from demonstrations provided fromboth learned agents and human experts.

 

Quick Read (beta)

loading the full paper ...