Abstract
We introduce ThreeDWorld (TDW), a platform for interactive multi-modalphysical simulation. With TDW, users can simulate high-fidelity sensory dataand physical interactions between mobile agents and objects in a wide varietyof rich 3D environments. TDW has several unique properties: 1) realtime nearphoto-realistic image rendering quality; 2) a library of objects andenvironments with materials for high-quality rendering, and routines enablinguser customization of the asset library; 3) generative procedures forefficiently building classes of new environments 4) high-fidelity audiorendering; 5) believable and realistic physical interactions for a wide varietyof material types, including cloths, liquid, and deformable objects; 6) a rangeof "avatar" types that serve as embodiments of AI agents, with the option foruser avatar customization; and 7) support for human interactions with VRdevices. TDW also provides a rich API enabling multiple agents to interactwithin a simulation and return a range of sensor and physics data representingthe state of the world. We present initial experiments enabled by the platformaround emerging research directions in computer vision, machine learning, andcognitive science, including multi-modal physical scene understanding,multi-agent interactions, models that "learn like a child", and attentionstudies in humans and neural networks. The simulation platform will be madepublicly available.