An Attention Free Transformer

  • 2021-05-28 20:45:30
  • Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, Josh Susskind
  • 190


We introduce Attention Free Transformer (AFT), an efficient variant ofTransformers that eliminates the need for dot product self attention. In an AFTlayer, the key and value are first combined with a set of learned positionbiases, the result of which is multiplied with the query in an element-wisefashion. This new operation has a memory complexity linear w.r.t. both thecontext size and the dimension of features, making it compatible to both largeinput and model sizes. We also introduce AFT-local and AFT-conv, two modelvariants that take advantage of the idea of locality and spatial weight sharingwhile maintaining global connectivity. We conduct extensive experiments on twoautoregressive modeling tasks (CIFAR10 and Enwik8) as well as an imagerecognition task (ImageNet-1K classification). We show that AFT demonstratescompetitive performance on all the benchmarks, while providing excellentefficiency at the same time.


Quick Read (beta)

loading the full paper ...