Abstract
Large language models (LLMs) have opened up new possibilities for intelligentagents, endowing them with human-like thinking and cognitive abilities. In thiswork, we delve into the potential of large language models (LLMs) in autonomousdriving (AD). We introduce DriveMLM, an LLM-based AD framework that can performclose-loop autonomous driving in realistic simulators. To this end, (1) webridge the gap between the language decisions and the vehicle control commandsby standardizing the decision states according to the off-the-shelf motionplanning module. (2) We employ a multi-modal LLM (MLLM) to model the behaviorplanning module of a module AD system, which uses driving rules, user commands,and inputs from various sensors (e.g., camera, lidar) as input and makesdriving decisions and provide explanations; This model can plug-and-play inexisting AD systems such as Apollo for close-loop driving. (3) We design aneffective data engine to collect a dataset that includes decision state andcorresponding explanation annotation for model training and evaluation. Weconduct extensive experiments and show that our model achieves 76.1 drivingscore on the CARLA Town05 Long, and surpasses the Apollo baseline by 4.7 pointsunder the same settings, demonstrating the effectiveness of our model. We hopethis work can serve as a baseline for autonomous driving with LLMs. Code andmodels shall be released at https://github.com/OpenGVLab/DriveMLM.