Abstract
Collecting and annotating real-world data for safety-critical physical AIsystems, such as Autonomous Vehicle (AV), is time-consuming and costly. It isespecially challenging to capture rare edge cases, which play a critical rolein training and testing of an AV system. To address this challenge, weintroduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipelinethat aims to generate challenging scenarios to facilitate downstream tasks suchas perception and driving policy training. Powering this pipeline isCosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundationmodel for the driving domain and are capable of controllable, high-fidelity,multi-view, and spatiotemporally consistent driving video generation. Weshowcase the utility of these models by applying Cosmos-Drive-Dreams to scalethe quantity and diversity of driving datasets with high-fidelity andchallenging scenarios. Experimentally, we demonstrate that our generated datahelps in mitigating long-tail distribution problems and enhances generalizationin downstream tasks such as 3D lane detection, 3D object detection and drivingpolicy learning. We open source our pipeline toolkit, dataset and model weightsthrough the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams