Abstract
We present Agent-to-Sim (ATS), a framework for learning interactive behaviormodels of 3D agents from casual longitudinal video collections. Different fromprior works that rely on marker-based tracking and multiview cameras, ATSlearns natural behaviors of animal and human agents non-invasively throughvideo observations recorded over a long time-span (e.g., a month) in a singleenvironment. Modeling 3D behavior of an agent requires persistent 3D tracking(e.g., knowing which point corresponds to which) over a long time period. Toobtain such data, we develop a coarse-to-fine registration method that tracksthe agent and the camera over time through a canonical 3D space, resulting in acomplete and persistent spacetime 4D representation. We then train a generativemodel of agent behaviors using paired data of perception and motion of an agentqueried from the 4D reconstruction. ATS enables real-to-sim transfer from videorecordings of an agent to an interactive behavior simulator. We demonstrateresults on pets (e.g., cat, dog, bunny) and human given monocular RGBD videoscaptured by a smartphone.