Abstract
Reinforcement learning for training end-to-end autonomous driving models inclosed-loop simulations is gaining growing attention. However, most simulationenvironments differ significantly from real-world conditions, creating asubstantial simulation-to-reality (sim2real) gap. To bridge this gap, someapproaches utilize scene reconstruction techniques to create photorealisticenvironments as a simulator. While this improves realistic sensor simulation,these methods are inherently constrained by the distribution of the trainingdata, making it difficult to render high-quality sensor data for noveltrajectories or corner case scenarios. Therefore, we propose ReconDreamer-RL, aframework designed to integrate video diffusion priors into scenereconstruction to aid reinforcement learning, thereby enhancing end-to-endautonomous driving training. Specifically, in ReconDreamer-RL, we introduceReconSimulator, which combines the video diffusion prior for appearancemodeling and incorporates a kinematic model for physical modeling, therebyreconstructing driving scenarios from real-world data. This narrows thesim2real gap for closed-loop evaluation and reinforcement learning. To covermore corner-case scenarios, we introduce the Dynamic Adversary Agent (DAA),which adjusts the trajectories of surrounding vehicles relative to the egovehicle, autonomously generating corner-case traffic scenarios (e.g., cut-in).Finally, the Cousin Trajectory Generator (CTG) is proposed to address the issueof training data distribution, which is often biased toward simplestraight-line movements. Experiments show that ReconDreamer-RL improvesend-to-end autonomous driving training, outperforming imitation learningmethods with a 5x reduction in the Collision Ratio.