Abstract
Large Language Models (LLMs) show growing promise in autonomous driving byreasoning over complex traffic scenarios to generate path plans. However, theirtendencies toward overconfidence, and hallucinations raise critical safetyconcerns. We introduce SafePath, a modular framework that augments LLM-basedpath planning with formal safety guarantees using conformal prediction.SafePath operates in three stages. In the first stage, we use an LLM thatgenerates a set of diverse candidate paths, exploring possible trajectoriesbased on agent behaviors and environmental cues. In the second stage, SafePathfilters out high-risk trajectories while guaranteeing that at least one safeoption is included with a user-defined probability, through a multiple-choicequestion-answering formulation that integrates conformal prediction. In thefinal stage, our approach selects the path with the lowest expected collisionrisk when uncertainty is low or delegates control to a human when uncertaintyis high. We theoretically prove that SafePath guarantees a safe trajectory witha user-defined probability, and we show how its human delegation rate can betuned to balance autonomy and safety. Extensive experiments on nuScenes andHighway-env show that SafePath reduces planning uncertainty by 77\% andcollision rates by up to 70\%, demonstrating effectiveness in making LLM-drivenpath planning more safer.