Vision Transformer with Deformable Attention

  • 2022-01-03 08:29:01
  • Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, Gao Huang
  • 45


Transformers have recently shown superior performances on various visiontasks. The large, sometimes even global, receptive field endows Transformermodels with higher representation power over their CNN counterparts.Nevertheless, simply enlarging receptive field also gives rise to severalconcerns. On the one hand, using dense attention e.g., in ViT, leads toexcessive memory and computational cost, and features can be influenced byirrelevant parts which are beyond the region of interests. On the other hand,the sparse attention adopted in PVT or Swin Transformer is data agnostic andmay limit the ability to model long range relations. To mitigate these issues,we propose a novel deformable self-attention module, where the positions of keyand value pairs in self-attention are selected in a data-dependent way. Thisflexible scheme enables the self-attention module to focus on relevant regionsand capture more informative features. On this basis, we present DeformableAttention Transformer, a general backbone model with deformable attention forboth image classification and dense prediction tasks. Extensive experimentsshow that our models achieve consistently improved results on comprehensivebenchmarks. Code is available at


Quick Read (beta)

loading the full paper ...