Abstract
Manually creating 3D environments for AR/VR applications is a complex processrequiring expert knowledge in 3D modeling software. Pioneering works facilitatethis process by generating room meshes conditioned on textual styledescriptions. Yet, many of these automatically generated 3D meshes do notadhere to typical room layouts, compromising their plausibility, e.g., byplacing several beds in one bedroom. To address these challenges, we presentControlRoom3D, a novel method to generate high-quality room meshes. Central toour approach is a user-defined 3D semantic proxy room that outlines a roughroom layout based on semantic bounding boxes and a textual description of theoverall room style. Our key insight is that when rendered to 2D, this 3Drepresentation provides valuable geometric and semantic information to controlpowerful 2D models to generate 3D consistent textures and geometry that alignswell with the proxy room. Backed up by an extensive study includingquantitative metrics and qualitative user evaluations, our method generatesdiverse and globally plausible 3D room meshes, thus empowering users to design3D rooms effortlessly without specialized knowledge.