End-to-end approaches to autonomous driving commonly rely on expertdemonstrations. Although humans are good drivers, they are not good coaches forend-to-end algorithms that demand dense on-policy supervision. On the contrary,automated experts that leverage privileged information can efficiently generatelarge scale on-policy and off-policy demonstrations. However, existingautomated experts for urban driving make heavy use of hand-crafted rules andperform suboptimally even on driving simulators, where ground-truth informationis available. To address these issues, we train a reinforcement learning expertthat maps bird's-eye view images to continuous low-level actions. While settinga new performance upper-bound on CARLA, our expert is also a better coach thatprovides informative supervision signals for imitation learning agents to learnfrom. Supervised by our reinforcement learning coach, a baseline end-to-endagent with monocular camera-input achieves expert-level performance. Ourend-to-end agent achieves a 78% success rate while generalizing to a new townand new weather on the NoCrash-dense benchmark and state-of-the-art performanceon the challenging public routes of the CARLA LeaderBoard.