Abstract
Task planning can require defining myriad domain knowledge about the world inwhich a robot needs to act. To ameliorate that effort, large language models(LLMs) can be used to score potential next actions during task planning, andeven generate action sequences directly, given an instruction in naturallanguage with no additional domain information. However, such methods eitherrequire enumerating all possible next steps for scoring, or generate free-formtext that may contain actions not possible on a given robot in its currentcontext. We present a programmatic LLM prompt structure that enables plangeneration functional across situated environments, robot capabilities, andtasks. Our key insight is to prompt the LLM with program-like specifications ofthe available actions and objects in an environment, as well as with exampleprograms that can be executed. We make concrete recommendations about promptstructure and generation constraints through ablation experiments, demonstratestate of the art success rates in VirtualHome household tasks, and deploy ourmethod on a physical robot arm for tabletop tasks. Website atprogprompt.github.io