Inverse Reinforcement Learning (IRL) is attractive in scenarios where rewardengineering can be tedious. However, prior IRL algorithms use on-policytransitions, which require intensive sampling from the current policy forstable and optimal performance. This limits IRL applications in the real world,where environment interactions can become highly expensive. To tackle thisproblem, we present Off-Policy Inverse Reinforcement Learning (OPIRL), which(1) adopts off-policy data distribution instead of on-policy and enablessignificant reduction of the number of interactions with the environment, (2)learns a stationary reward function that is transferable with highgeneralization capabilities on changing dynamics, and (3) leveragesmode-covering behavior for faster convergence. We demonstrate that our methodis considerably more sample efficient and generalizes to novel environmentsthrough the experiments. Our method achieves better or comparable results onpolicy performance baselines with significantly fewer interactions.Furthermore, we empirically show that the recovered reward function generalizesto different tasks where prior arts are prone to fail.