Abstract
Data-driven character animation based on motion capture can produce highlynaturalistic behaviors and, when combined with physics simulation, can providefor natural procedural responses to physical perturbations, environmentalchanges, and morphological discrepancies. Motion capture remains the mostpopular source of motion data, but collecting mocap data typically requiresheavily instrumented environments and actors. In this paper, we propose amethod that enables physically simulated characters to learn skills from videos(SFV). Our approach, based on deep pose estimation and deep reinforcementlearning, allows data-driven animation to leverage the abundance of publiclyavailable video clips from the web, such as those from YouTube. This has thepotential to enable fast and easy design of character controllers simply byquerying for video recordings of the desired behavior. The resultingcontrollers are robust to perturbations, can be adapted to new settings, canperform basic object interactions, and can be retargeted to new morphologiesvia reinforcement learning. We further demonstrate that our method can predictpotential human motions from still images, by forward simulation of learnedcontrollers initialized from the observed pose. Our framework is able to learna broad range of dynamic skills, including locomotion, acrobatics, and martialarts.