Abstract
In inverse reinforcement learning (IRL), an agent seeks to replicate expertdemonstrations through interactions with the environment. Traditionally, IRL istreated as an adversarial game, where an adversary searches over reward models,and a learner optimizes the reward through repeated RL procedures. Thisgame-solving approach is both computationally expensive and difficult tostabilize. In this work, we propose a novel approach to IRL by direct policyoptimization: exploiting a linear factorization of the return as the innerproduct of successor features and a reward vector, we design an IRL algorithmby policy gradient descent on the gap between the learner and expert features.Our non-adversarial method does not require learning a reward function and canbe solved seamlessly with existing actor-critic RL algorithms. Remarkably, ourapproach works in state-only settings without expert action labels, a settingwhich behavior cloning (BC) cannot solve. Empirical results demonstrate thatour method learns from as few as a single expert demonstration and achievesimproved performance on various control tasks.