Abstract
We present Large Language Model for Mixed Reality (LLMR), a framework for thereal-time creation and modification of interactive Mixed Reality experiencesusing LLMs. LLMR leverages novel strategies to tackle difficult cases whereideal training data is scarce, or where the design goal requires the synthesisof internal dynamics, intuitive analysis, or advanced interactivity. Ourframework relies on text interaction and the Unity game engine. Byincorporating techniques for scene understanding, task planning,self-debugging, and memory management, LLMR outperforms the standard GPT-4 by4x in average error rate. We demonstrate LLMR's cross-platform interoperabilitywith several example worlds, and evaluate it on a variety of creation andmodification tasks to show that it can produce and edit diverse objects, tools,and scenes. Finally, we conducted a usability study (N=11) with a diverse setthat revealed participants had positive experiences with the system and woulduse it again.