Abstract
Large language models (LLMs) have shown continuously improving multilingualcapabilities, and even small-scale open-source models have demonstrated rapidperformance enhancement. In this paper, we systematically explore the abilitiesof open LLMs with less than ten billion parameters to handle multilingualmachine translation (MT) tasks. We conduct comprehensive evaluations on sixpopular LLMs and find that models like Gemma2-9B exhibit impressivemultilingual translation capabilities. We then introduce the Parallel-FirstMonolingual-Second (PFMS) data mixing strategy in the continual pretrainingstage to further enhance the MT performance and present GemmaX2-28, a 9B modelachieving top-tier multilingual translation performance across 28 languages.Specifically, GemmaX2-28 consistently outperforms the state-of-the-art (SOTA)models such as TowerInstruct and XALMA and achieves competitive performancewith Google Translate and GPT-4-turbo.