Abstract
We present a novel approach to automatically synthesize "wayfindinginstructions" for an embodied robot agent. In contrast to prior approaches thatare heavily reliant on human-annotated datasets designed exclusively forspecific simulation platforms, our algorithm uses in-context learning tocondition an LLM to generate instructions using just a few references. Using anLLM-based Visual Question Answering strategy, we gather detailed informationabout the environment which is used by the LLM for instruction synthesis. Weimplement our approach on multiple simulation platforms including Matterport3D,AI Habitat and ThreeDWorld, thereby demonstrating its platform-agnostic nature.We subjectively evaluate our approach via a user study and observe that 83.3%of users find the synthesized instructions accurately capture the details ofthe environment and show characteristics similar to those of human-generatedinstructions. Further, we conduct zero-shot navigation with multiple approacheson the REVERIE dataset using the generated instructions, and observe very closecorrelation with the baseline on standard success metrics (< 1% change in SR),quantifying the viability of generated instructions in replacinghuman-annotated data. We finally discuss the applicability of our approach inenabling a generalizable evaluation of embodied navigation policies. To thebest of our knowledge, ours is the first LLM-driven approach capable ofgenerating "human-like" instructions in a platform-agnostic manner, withouttraining.