Abstract
Large language models (LLMs) have demonstrated remarkable performance,particularly in multilingual contexts. While recent studies suggest that LLMscan transfer skills learned in one language to others, the internal mechanismsbehind this ability remain unclear. We observed that the neuron activationpatterns of LLMs exhibit similarities when processing the same language,revealing the existence and location of key linguistic regions. Additionally,we found that neuron activation patterns are similar when processing sentenceswith the same semantic meaning in different languages. This indicates that LLMsmap semantically identical inputs from different languages into a "LinguaFranca", a common semantic latent space that allows for consistent processingacross languages. This semantic alignment becomes more pronounced with trainingand increased model size, resulting in a more language-agnostic activationpattern. Moreover, we found that key linguistic neurons are concentrated in thefirst and last layers of LLMs, becoming denser in the first layers as trainingprogresses. Experiments on BLOOM and LLaMA2 support these findings,highlighting the structural evolution of multilingual LLMs during training andscaling up. This paper provides insights into the internal workings of LLMs,offering a foundation for future improvements in their cross-lingualcapabilities.