Abstract
Planning has been part of the core pursuit for artificial intelligence sinceits conception, but earlier AI agents mostly focused on constrained settingsbecause many of the cognitive substrates necessary for human-level planninghave been lacking. Recently, language agents powered by large language models(LLMs) have shown interesting capabilities such as tool use and reasoning. Arethese language agents capable of planning in more complex settings that are outof the reach of prior AI agents? To advance this investigation, we proposeTravelPlanner, a new planning benchmark that focuses on travel planning, acommon real-world planning scenario. It provides a rich sandbox environment,various tools for accessing nearly four million data records, and 1,225meticulously curated planning intents and reference plans. Comprehensiveevaluations show that the current language agents are not yet capable ofhandling such complex planning tasks-even GPT-4 only achieves a success rate of0.6%. Language agents struggle to stay on task, use the right tools to collectinformation, or keep track of multiple constraints. However, we note that themere possibility for language agents to tackle such a complex problem is initself non-trivial progress. TravelPlanner provides a challenging yetmeaningful testbed for future language agents.