Abstract
The success of multilingual pre-trained models is underpinned by theirability to learn representations shared by multiple languages even in absenceof any explicit supervision. However, it remains unclear how these models learnto generalise across languages. In this work, we conjecture that multilingualpre-trained models can derive language-universal abstractions about grammar. Inparticular, we investigate whether morphosyntactic information is encoded inthe same subset of neurons in different languages. We conduct the firstlarge-scale empirical study over 43 languages and 14 morphosyntactic categorieswith a state-of-the-art neuron-level probe. Our findings show that thecross-lingual overlap between neurons is significant, but its extent may varyacross categories and depends on language proximity and pre-training data size.