Do Transformer Modifications Transfer Across Implementations and Applications?

  • 2021-02-23 22:44:54
  • Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
  • 136

Abstract

The research community has proposed copious modifications to the Transformerarchitecture since it was introduced over three years ago, relatively few ofwhich have seen widespread adoption. In this paper, we comprehensively evaluatemany of these modifications in a shared experimental setting that covers mostof the common uses of the Transformer in natural language processing.Surprisingly, we find that most modifications do not meaningfully improveperformance. Furthermore, most of the Transformer variants we found beneficialwere either developed in the same codebase that we used or are relatively minorchanges. We conjecture that performance improvements may strongly depend onimplementation details and correspondingly make some recommendations forimproving the generality of experimental results.

 

Quick Read (beta)

loading the full paper ...