Abstract
Reinforcement Learning (RL) has the potential to surpass human performance indriving without needing any expert supervision. Despite its promise, thestate-of-the-art in sensorimotor self-driving is dominated by imitationlearning methods due to the inherent shortcomings of RL algorithms.Nonetheless, RL agents are able to discover highly successful policies whenprovided with privileged ground truth representations of the environment. Inthis work, we investigate what separates privileged RL agents from sensorimotoragents for urban driving in order to bridge the gap between the two. We proposevision-based deep learning models to approximate the privileged representationsfrom sensor data. In particular, we identify aspects of state representationthat are crucial for the success of the RL agent such as desired routegeneration and stop zone prediction, and propose solutions to gradually developless privileged RL agents. We also observe that bird's-eye-view models trainedon offline datasets do not generalize to online RL training due to distributionmismatch. Through rigorous evaluation on the CARLA simulation environment, weshed light on the significance of the state representations in RL forautonomous driving and point to unresolved challenges for future research.