Abstract
This paper presents an effective approach for learning novel 4D embodiedworld models, which predict the dynamic evolution of 3D scenes over time inresponse to an embodied agent's actions, providing both spatial and temporalconsistency. We propose to learn a 4D world model by training on RGB-DN (RGB,Depth, and Normal) videos. This not only surpasses traditional 2D models byincorporating detailed shape, configuration, and temporal changes into theirpredictions, but also allows us to effectively learn accurate inverse dynamicmodels for an embodied agent. Specifically, we first extend existing roboticmanipulation video datasets with depth and normal information leveragingoff-the-shelf models. Next, we fine-tune a video generation model on thisannotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) foreach frame. We then present an algorithm to directly convert generated RGB,Depth, and Normal videos into a high-quality 4D scene of the world. Our methodensures temporal and spatial coherence in 4D scene predictions from embodiedscenarios, enables novel view synthesis for embodied environments, andfacilitates policy learning that significantly outperforms those derived fromprior video-based world models.