Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

  • 2021-09-14 16:07:09
  • Mozhdeh Gheini, Xiang Ren, Jonathan May
  • 0


We study the power of cross-attention in the Transformer architecture withinthe context of transfer learning for machine translation, and extend thefindings of studies into cross-attention when training from scratch. We conducta series of experiments through fine-tuning a translation model on data whereeither the source or target language has changed. These experiments reveal thatfine-tuning only the cross-attention parameters is nearly as effective asfine-tuning all parameters (i.e., the entire translation model). We provideinsights into why this is the case and observe that limiting fine-tuning inthis manner yields cross-lingually aligned embeddings. The implications of thisfinding for researchers and practitioners include a mitigation of catastrophicforgetting, the potential for zero-shot translation, and the ability to extendmachine translation models to several new language pairs with reduced parameterstorage overhead.


Quick Read (beta)

loading the full paper ...