Abstract
Multilingual Large Language Models (LLMs) develop cross-lingual abilitiesdespite being trained on limited parallel data. However, they often struggle togenerate responses in the intended language, favoring high-resource languagessuch as English. In this work, we introduce CoCo-CoLa (Correct Concept -Correct Language), a novel metric to evaluate language adherence inmultilingual LLMs. Using fine-tuning experiments on a closed-book QA taskacross seven languages, we analyze how training in one language affects others'performance. Our findings reveal that multilingual models share task knowledgeacross languages but exhibit biases in the selection of output language. Weidentify language-specific layers, showing that final layers play a crucialrole in determining output language. Accordingly, we propose a partial trainingstrategy that selectively fine-tunes key layers, improving language adherencewhile significantly reducing computational cost. Our method achieves comparableor superior performance to full fine-tuning, particularly for low-resourcelanguages, offering a more efficient multilingual adaptation.