Abstract
Modern approaches to autonomous driving rely heavily on learned componentstrained with large amounts of human driving data via imitation learning.However, these methods require large amounts of expensive data collection andeven then face challenges with safely handling long-tail scenarios andcompounding errors over time. At the same time, pure Reinforcement Learning(RL) methods can fail to learn performant policies in sparse, constrained, andchallenging-to-define reward settings such as autonomous driving. Both of thesechallenges make deploying purely cloned or pure RL policies in safety criticalapplications such as autonomous vehicles challenging. In this paper we proposeCombining IMitation and Reinforcement Learning (CIMRL) approach - a safereinforcement learning framework that enables training driving policies insimulation through leveraging imitative motion priors and safety constraints.CIMRL does not require extensive reward specification and improves on theclosed loop behavior of pure cloning methods. By combining RL and imitation, wedemonstrate that our method achieves state-of-the-art results in closed loopsimulation and real world driving benchmarks.