Reinforcement learning has been demonstrated to outperform even the besthumans in complex domains like video games. However, running reinforcementlearning experiments on the required scale for autonomous driving is extremelydifficult. Building a large scale reinforcement learning system anddistributing it across many GPUs is challenging. Gathering experience duringtraining on real world vehicles is prohibitive from a safety and scalabilityperspective. Therefore, an efficient and realistic driving simulator isrequired that uses a large amount of data from real-world driving. We bringthese capabilities together and conduct large-scale reinforcement learningexperiments for autonomous driving. We demonstrate that our policy performanceimproves with increasing scale. Our best performing policy reduces the failurerate by 64% while improving the rate of driving progress by 25% compared to thepolicies produced by state-of-the-art machine learning for autonomous driving.