Abstract
Multilingual Neural Machine Translation (NMT) models have yielded largeempirical success in transfer learning settings. However, these black-boxrepresentations are poorly understood, and their mode of transfer remainselusive. In this work, we attempt to understand massively multilingual NMTrepresentations (with 103 languages) using Singular Value Canonical CorrelationAnalysis (SVCCA), a representation similarity framework that allows us tocompare representations across different languages, layers and models. Ouranalysis validates several empirical results and long-standing intuitions, andunveils new observations regarding how representations evolve in a multilingualtranslation model. We draw three major conclusions from our analysis, withimplications on cross-lingual transfer learning: (i) Encoder representations ofdifferent languages cluster based on linguistic similarity, (ii)Representations of a source language learned by the encoder are dependent onthe target language, and vice-versa, and (iii) Representations of high resourceand/or linguistically similar languages are more robust when fine-tuning on anarbitrary language pair, which is critical to determining how muchcross-lingual transfer can be expected in a zero or few-shot setting. Wefurther connect our findings with existing empirical observations inmultilingual NMT and transfer learning.