Abstract
Transformers have been successfully applied in the field of video-based 3Dhuman pose estimation. However, the high computational costs of these videopose transformers (VPTs) make them impractical on resource-constrained devices.In this paper, we present a hierarchical plug-and-play pruning-and-recoveringframework, called Hierarchical Hourglass Tokenizer (H$_{2}$OT), for efficienttransformer-based 3D human pose estimation from videos. H$_{2}$OT begins withprogressively pruning pose tokens of redundant frames and ends with recoveringfull-length sequences, resulting in a few pose tokens in the intermediatetransformer blocks and thus improving the model efficiency. It works with twokey modules, namely, a Token Pruning Module (TPM) and a Token Recovering Module(TRM). TPM dynamically selects a few representative tokens to eliminate theredundancy of video frames, while TRM restores the detailed spatio-temporalinformation based on the selected tokens, thereby expanding the network outputto the original full-length temporal resolution for fast inference. Our methodis general-purpose: it can be easily incorporated into common VPT models onboth seq2seq and seq2frame pipelines while effectively accommodating differenttoken pruning and recovery strategies. In addition, our H$_{2}$OT reveals thatmaintaining the full pose sequence is unnecessary, and a few pose tokens ofrepresentative frames can achieve both high efficiency and estimation accuracy.Extensive experiments on multiple benchmark datasets demonstrate both theeffectiveness and efficiency of the proposed method. Code and models areavailable at https://github.com/NationalGAILab/HoT.