Multiview Transformers for Video Recognition

  • 2022-01-12 03:33:57
  • Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, Cordelia Schmid
  • 14

Abstract

Video understanding requires reasoning at multiple spatiotemporal resolutions-- from short fine-grained motions to events taking place over longerdurations. Although transformer architectures have recently advanced thestate-of-the-art, they have not explicitly modelled different spatiotemporalresolutions. To this end, we present Multiview Transformers for VideoRecognition (MTV). Our model consists of separate encoders to representdifferent views of the input video with lateral connections to fuse informationacross views. We present thorough ablation studies of our model and show thatMTV consistently performs better than single-view counterparts in terms ofaccuracy and computational cost across a range of model sizes. Furthermore, weachieve state-of-the-art results on five standard datasets, and improve evenfurther with large-scale pretraining. We will release code and pretrainedcheckpoints.

 

Quick Read (beta)

loading the full paper ...