Abstract
Human videos offer a scalable way to train robot manipulation policies, butlack the action labels needed by standard imitation learning algorithms.Existing cross-embodiment approaches try to map human motion to robot actions,but often fail when the embodiments differ significantly. We propose X-Sim, areal-to-sim-to-real framework that uses object motion as a dense andtransferable signal for learning robot policies. X-Sim starts by reconstructinga photorealistic simulation from an RGBD human video and tracking objecttrajectories to define object-centric rewards. These rewards are used to traina reinforcement learning (RL) policy in simulation. The learned policy is thendistilled into an image-conditioned diffusion policy using synthetic rolloutsrendered with varied viewpoints and lighting. To transfer to the real world,X-Sim introduces an online domain adaptation technique that aligns real andsimulated observations during deployment. Importantly, X-Sim does not requireany robot teleoperation data. We evaluate it across 5 manipulation tasks in 2environments and show that it: (1) improves task progress by 30% on averageover hand-tracking and sim-to-real baselines, (2) matches behavior cloning with10x less data collection time, and (3) generalizes to new camera viewpoints andtest-time changes. Code and videos are available athttps://portal-cornell.github.io/X-Sim/.