Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

  • 2021-06-09 17:59:34
  • Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, Xiaolong Wang
  • 0


Estimating 3D hand and object pose from a single image is an extremelychallenging problem: hands and objects are often self-occluded duringinteractions, and the 3D annotations are scarce as even humans cannot directlylabel the ground-truths from a single image perfectly. To tackle thesechallenges, we propose a unified framework for estimating the 3D hand andobject poses with semi-supervised learning. We build a joint learning frameworkwhere we perform explicit contextual reasoning between hand and objectrepresentations by a Transformer. Going beyond limited 3D annotations in asingle image, we leverage the spatial-temporal consistency in large-scalehand-object videos as a constraint for generating pseudo labels insemi-supervised learning. Our method not only improves hand pose estimation inchallenging real-world dataset, but also substantially improve the object posewhich has fewer ground-truths per instance. By training with large-scalediverse videos, our model also generalizes better across multiple out-of-domaindatasets. Project page and code:


Quick Read (beta)

loading the full paper ...