Abstract
Real-time tracking of human body motion is crucial for interactive andimmersive experiences in AR/VR. However, very limited sensor data about thebody is available from standalone wearable devices such as HMDs (Head MountedDevices) or AR glasses. In this work, we present a reinforcement learningframework that takes in sparse signals from an HMD and two controllers, andsimulates plausible and physically valid full body motions. Using high qualityfull body motion as dense supervision during training, a simple policy networkcan learn to output appropriate torques for the character to balance, walk, andjog, while closely following the input signals. Our results demonstratesurprisingly similar leg motions to ground truth without any observations ofthe lower body, even when the input is only the 6D transformations of the HMD.We also show that a single policy can be robust to diverse locomotion styles,different body sizes, and novel environments.