Abstract
Understanding, navigating, and exploring the 3D physical real world has longbeen a central challenge in the development of artificial intelligence. In thiswork, we take a step toward this goal by introducing GenEx, a system capable ofplanning complex embodied world exploration, guided by its generativeimagination that forms priors (expectations) about the surroundingenvironments. GenEx generates an entire 3D-consistent imaginative environmentfrom as little as a single RGB image, bringing it to life through panoramicvideo streams. Leveraging scalable 3D world data curated from Unreal Engine,our generative model is rounded in the physical world. It captures a continuous360-degree environment with little effort, offering a boundless landscape forAI agents to explore and interact with. GenEx achieves high-quality worldgeneration, robust loop consistency over long trajectories, and demonstratesstrong 3D capabilities such as consistency and active 3D mapping. Powered bygenerative imagination of the world, GPT-assisted agents are equipped toperform complex embodied tasks, including both goal-agnostic exploration andgoal-driven navigation. These agents utilize predictive expectation regardingunseen parts of the physical world to refine their beliefs, simulate differentoutcomes based on potential decisions, and make more informed choices. Insummary, we demonstrate that GenEx provides a transformative platform foradvancing embodied AI in imaginative spaces and brings potential for extendingthese capabilities to real-world exploration.