Abstract
Recent research has been increasingly focusing on developing 3D world modelsthat simulate complex real-world scenarios. World models have found broadapplications across various domains, including embodied AI, autonomous driving,entertainment, etc. A more realistic simulation with accurate physics willeffectively narrow the sim-to-real gap and allow us to gather rich informationabout the real world conveniently. While traditional manual modeling hasenabled the creation of virtual 3D scenes, modern approaches have leveragedadvanced machine learning algorithms for 3D world generation, with most recentadvances focusing on generative methods that can create virtual worlds based onuser instructions. This work explores such a research direction by proposingLatticeWorld, a simple yet effective 3D world generation framework thatstreamlines the industrial production pipeline of 3D environments. LatticeWorldleverages lightweight LLMs (LLaMA-2-7B) alongside the industry-grade renderingengine (e.g., Unreal Engine 5) to generate a dynamic environment. Our proposedframework accepts textual descriptions and visual instructions as multimodalinputs and creates large-scale 3D interactive worlds with dynamic agents,featuring competitive multi-agent interaction, high-fidelity physicssimulation, and real-time rendering. We conduct comprehensive experiments toevaluate LatticeWorld, showing that it achieves superior accuracy in scenelayout generation and visual fidelity. Moreover, LatticeWorld achieves over a$90\times$ increase in industrial production efficiency while maintaining highcreative quality compared with traditional manual production methods. Our demovideo is available at https://youtu.be/8VWZXpERR18