Abstract
Large Language Models (LLMs) and Vision Language Models (VLMs) have beenwidely used for embodied symbolic planning. Yet, how to effectively use thesemodels for closed-loop symbolic planning remains largely unexplored. Becausethey operate as black boxes, LLMs and VLMs can produce unpredictable or costlyerrors, making their use in high-level robotic planning especially challenging.In this work, we investigate how to use VLMs as closed-loop symbolic plannersfor robotic applications from a control-theoretic perspective. Concretely, westudy how the control horizon and warm-starting impact the performance of VLMsymbolic planners. We design and conduct controlled experiments to gaininsights that are broadly applicable to utilizing VLMs as closed-loop symbolicplanners, and we discuss recommendations that can help improve the performanceof VLM symbolic planners.