Abstract
Automatic creation of 3D scenes for immersive VR presence has been asignificant research focus for decades. However, existing methods often rely oneither high-poly mesh modeling with post-hoc simplification or massive 3DGaussians, resulting in a complex pipeline or limited visual realism. In thispaper, we demonstrate that such exhaustive modeling is unnecessary forachieving compelling immersive experience. We introduce ImmerseGen, a novelagent-guided framework for compact and photorealistic world modeling.ImmerseGen represents scenes as hierarchical compositions of lightweightgeometric proxies, i.e., simplified terrain and billboard meshes, and generatesphotorealistic appearance by synthesizing RGBA textures onto these proxies.Specifically, we propose terrain-conditioned texturing for user-centric baseworld synthesis, and RGBA asset texturing for midground and foreground scenery.This reformulation offers several advantages: (i) it simplifies modeling byenabling agents to guide generative models in producing coherent textures thatintegrate seamlessly with the scene; (ii) it bypasses complex geometry creationand decimation by directly synthesizing photorealistic textures on proxies,preserving visual quality without degradation; (iii) it enables compactrepresentations suitable for real-time rendering on mobile VR headsets. Toautomate scene creation from text prompts, we introduce VLM-based modelingagents enhanced with semantic grid-based analysis for improved spatialreasoning and accurate asset placement. ImmerseGen further enriches scenes withdynamic effects and ambient audio to support multisensory immersion.Experiments on scene generation and live VR showcases demonstrate thatImmerseGen achieves superior photorealism, spatial coherence and renderingefficiency compared to prior methods. Project webpage:https://immersegen.github.io.