Abstract
To democratize large language models (LLMs) to most natural languages, it isimperative to make these models capable of understanding and generating textsin many languages, in particular low-resource ones. While recent multilingualLLMs demonstrate remarkable performance in such capabilities, these LLMs stillsupport a limited number of human languages due to the lack of training datafor low-resource languages. Moreover, these LLMs are not yet aligned with humanpreference for downstream tasks, which is crucial for the success of LLMs inEnglish. In this paper, we introduce xLLaMA-100 and xBLOOM-100 (collectivelyxLLMs-100), which scale the multilingual capabilities of LLaMA and BLOOM to 100languages. To do so, we construct two datasets: a multilingual instructiondataset including 100 languages, which represents the largest language coverageto date, and a cross-lingual human feedback dataset encompassing 30 languages.We perform multilingual instruction tuning on the constructed instruction dataand further align the LLMs with human feedback using the DPO algorithm on ourcross-lingual human feedback dataset. We evaluate the multilingualunderstanding and generating capabilities of xLLMs-100 on five multilingualbenchmarks. Experimental results show that xLLMs-100 consistently outperformsits peers across the benchmarks by considerable margins, defining a newstate-of-the-art multilingual LLM that supports 100 languages.