Abstract
Large-scale video generative models can synthesize diverse and realisticvisual content for dynamic world creation, but they often lack element-wisecontrollability, hindering their use in editing scenes and training embodied AIagents. We propose Dreamland, a hybrid world generation framework combining thegranular control of a physics-based simulator and the photorealistic contentoutput of large-scale pretrained generative models. In particular, we design alayered world abstraction that encodes both pixel-level and object-levelsemantics and geometry as an intermediate representation to bridge thesimulator and the generative model. This approach enhances controllability,minimizes adaptation cost through early alignment with real-worlddistributions, and supports off-the-shelf use of existing and future pretrainedgenerative models. We further construct a D3Sim dataset to facilitate thetraining and evaluation of hybrid generation pipelines. Experiments demonstratethat Dreamland outperforms existing baselines with 50.8% improved imagequality, 17.9% stronger controllability, and has great potential to enhanceembodied agent training. Code and data will be made available.