Abstract
Large language models (LLMs) demonstrate remarkable performance across aspectrum of languages. In this work, we delve into the question: How do LLMshandle multilingualism? We introduce a framework that depicts LLMs' processingof multilingual inputs: In the first several layers, LLMs understand thequestion, converting multilingual inputs into English to facilitate thetask-solving phase. In the intermediate layers, LLMs engage in problem-solvingby thinking in English and incorporating multilingual knowledge to obtainfactual content, leveraging the self-attention and feed-forward structures,respectively. In the last several layers, LLMs generate responses that alignwith the original language of the query. In addition, we investigate theexistence of language-specific neurons when processing a certain language. Todetect neurons activated by the input language, even without labels, weinnovatively design a Parallel Language specific Neuron Detection($\texttt{PLND}$) method that effectively measures the significance of neuronswhen handling multilingual inputs. By comprehensive ablation analysis throughdeactivating neurons of different layers and structures, we verify theframework that we propose. Additionally, we demonstrate that we can utilizesuch a framework to effectively enhance the multilingual ability with much lesstraining effort.