Abstract
Large Language Models (LLMs) have demonstrated remarkable multilingualcapabilities, yet challenges persist in adapting these models for low-resourcelanguages. In this study, we investigate the effects of Low-Rank Adaptation(LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models forMarathi, a language with limited resources. Using a translated Alpaca datasetwith 52,000 instruction-response pairs, our findings reveal that whileevaluation metrics often show a performance decline post-fine-tuning, manualassessments frequently suggest that the fine-tuned models outperform theiroriginal counterparts. The observations indicate improvements in targetlanguage generation capabilities but a reduction in reasoning abilitiesfollowing language adaptation. These results underscore the need for improvedevaluation methodologies and the creation of high-quality native datasets toaccurately assess language-specific model performance in low-resource settings.