MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

  • 2021-11-24 18:59:02
  • Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, Luc Van Gool
  • 0


Estimating 3D human poses from monocular videos is a challenging task due todepth ambiguity and self-occlusion. Most existing works attempt to solve bothissues by exploiting spatial and temporal relationships. However, those worksignore the fact that it is an inverse problem where multiple feasible solutions(i.e., hypotheses) exist. To relieve this limitation, we propose aMulti-Hypothesis Transformer (MHFormer) that learns spatio-temporalrepresentations of multiple plausible pose hypotheses. In order to effectivelymodel multi-hypothesis dependencies and build strong relationships acrosshypothesis features, the task is decomposed into three stages: (i) Generatemultiple initial hypothesis representations; (ii) Model self-hypothesiscommunication, merge multiple hypotheses into a single converged representationand then partition it into several diverged hypotheses; (iii) Learncross-hypothesis communication and aggregate the multi-hypothesis features tosynthesize the final 3D pose. Through the above processes, the finalrepresentation is enhanced and the synthesized pose is much more accurate.Extensive experiments show that MHFormer achieves state-of-the-art results ontwo challenging datasets: Human3.6M and MPI-INF-3DHP. Without bells andwhistles, its performance surpasses the previous best result by a large marginof 3% on Human3.6M. Code and models are available at


Quick Read (beta)

loading the full paper ...