Abstract
Multilingual large language models (LLMs) aim towards robust natural languageunderstanding across diverse languages, yet their performance significantlydegrades on low-resource languages. This work explores whether existingtechniques to identify language-specific neurons can be leveraged to enhancecross-lingual task performance of lowresource languages. We conduct detailedexperiments covering existing language-specific neuron identificationtechniques (such as Language Activation Probability Entropy and activationprobability-based thresholding) and neuron-specific LoRA fine-tuning withmodels like Llama 3.1 and Mistral Nemo. We find that such neuron-specificinterventions are insufficient to yield cross-lingual improvements ondownstream tasks (XNLI, XQuAD) in lowresource languages. This study highlightsthe challenges in achieving cross-lingual generalization and provides criticalinsights for multilingual LLMs.