Abstract
Large Language Models (LLMs) have been adopted and deployed worldwide for abroad variety of applications. However, ensuring their safe use remains asignificant challenge. Preference training and safety measures often overfit toharms prevalent in Western-centric datasets, and safety protocols frequentlyfail to extend to multilingual settings. In this work, we explore model mergingin a diverse multi-task setting, combining safety and general-purpose taskswithin a multilingual context. Each language introduces unique and variedlearning challenges across tasks. We find that objective-based merging is moreeffective than mixing data, with improvements of up to 8% and 10% in generalperformance and safety respectively. We also find that language-based mergingis highly effective -- by merging monolingually fine-tuned models, we achieve a4% increase in general performance and 7% reduction in harm across alllanguages on top of the data mixtures method using the same available data.Overall, our comprehensive study of merging approaches provides a usefulframework for building strong and safe multilingual models.