Abstract
Parameter Efficient Finetuning (PEFT) has emerged as a viable solution forimproving the performance of Large Language Models (LLMs) without requiringmassive resources and compute. Prior work on multilingual evaluation has shownthat there is a large gap between the performance of LLMs on English and otherlanguages. Further, there is also a large gap between the performance ofsmaller open-source models and larger LLMs. Finetuning can be an effective wayto bridge this gap and make language models more equitable. In this work, wefinetune the LLama-2-7B and Mistral-7B models on two synthetic multilingualinstruction tuning datasets to determine its effect on model performance on sixdownstream tasks covering forty languages in all. Additionally, we experimentwith various parameters, such as rank for low-rank adaptation and values ofquantisation to determine their effects on downstream performance and find thathigher rank and higher quantisation values benefit low-resource languages. Wefind that PEFT of smaller open-source models sometimes bridges the gap betweenthe performance of these models and the larger ones, however, Englishperformance can take a hit. We also find that finetuning sometimes improvesperformance on low-resource languages, while degrading performance onhigh-resource languages.