Abstract
It is now established that modern neural language models can be successfullytrained on multiple languages simultaneously without changes to the underlyingarchitecture, providing an easy way to adapt a variety of NLP models tolow-resource languages. But what kind of knowledge is really shared amonglanguages within these models? Does multilingual training mostly lead to analignment of the lexical representation spaces or does it also enable thesharing of purely grammatical knowledge? In this paper we dissect differentforms of cross-lingual transfer and look for its most determining factors,using a variety of models and probing tasks. We find that exposing our languagemodels to a related language does not always increase grammatical knowledge inthe target language, and that optimal conditions for lexical-semantic transfermay not be optimal for syntactic transfer.