Abstract
Multilingual Alignment is an effective and representative paradigm to enhanceLLMs' multilingual capabilities, which transfers the capabilities from thehigh-resource languages to the low-resource languages. Meanwhile, someresearches on language-specific neurons reveal that there are language-specificneurons that are selectively activated in LLMs when processing differentlanguages. This provides a new perspective to analyze and understand LLMs'mechanisms more specifically in multilingual scenarios. In this work, wepropose a new finer-grained neuron identification algorithm, which detectslanguage neurons~(including language-specific neurons and language-relatedneurons) and language-agnostic neurons. Furthermore, based on thedistributional characteristics of different types of neurons, we divide theLLMs' internal process for multilingual inference into four parts: (1)multilingual understanding, (2) shared semantic space reasoning, (3)multilingual output space transformation, and (4) vocabulary space outputting.Additionally, we systematically analyze the models before and after alignmentwith a focus on different types of neurons. We also analyze the phenomenon of''Spontaneous Multilingual Alignment''. Overall, our work conducts acomprehensive investigation based on different types of neurons, providingempirical results and valuable insights for better understanding multilingualalignment and multilingual capabilities of LLMs.