Abstract
Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL)search over candidate reward functions and solve a reinforcement learningproblem in the inner loop. This creates a rather strange inversion where aharder problem, reinforcement learning, is in the inner loop of a presumablyeasier problem, imitation learning. In this work, we show that betterutilization of expert demonstrations can reduce the need for hard explorationin the inner RL loop, hence accelerating learning. Specifically, we propose twosimple recipes: (1) placing expert transitions into the replay buffer of theinner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learnerabout high reward states instead of forcing the learner to discover themthrough extensive exploration, and (2) using expert actions in Q valuebootstrapping in order to improve the target Q value estimates and moreaccurately describe high value expert states. Our methods show significantgains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks,speeding up recovery to 70\% of deterministic expert performance by 2.13x onHalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.