A Survey of Transformers

  • 2021-06-08 17:43:08
  • Tianyang Lin, Yuxin Wang, Xiangyang Liu, Xipeng Qiu
  • 67

Abstract

Transformers have achieved great success in many artificial intelligencefields, such as natural language processing, computer vision, and audioprocessing. Therefore, it is natural to attract lots of interest from academicand industry researchers. Up to the present, a great variety of Transformervariants (a.k.a. X-formers) have been proposed, however, a systematic andcomprehensive literature review on these Transformer variants is still missing.In this survey, we provide a comprehensive review of various X-formers. Wefirst briefly introduce the vanilla Transformer and then propose a new taxonomyof X-formers. Next, we introduce the various X-formers from three perspectives:architectural modification, pre-training, and applications. Finally, we outlinesome potential directions for future research.

 

Quick Read (beta)

loading the full paper ...