Abstract
Existing research predominantly focuses on developing powerful languagelearning models (LLMs) for mathematical reasoning within monolingual languages,with few explorations in preserving efficacy in a multilingual context. Tobridge this gap, this paper pioneers exploring and training powerfulMultilingual Math Reasoning (xMR) LLMs. Firstly, by utilizing translation, weconstruct the first multilingual math reasoning instruction dataset,MGSM8KInstruct, encompassing ten distinct languages, thus addressing the issueof training data scarcity in xMR tasks. Based on the collected dataset, wepropose different training strategies to build powerful xMR LLMs, namedMathOctopus, notably outperform conventional open-source LLMs and exhibitsuperiority over ChatGPT in few-shot scenarios. Notably, MathOctopus-13Breaches 47.6% accuracy which exceeds ChatGPT 46.3% on MGSM testset. Beyondremarkable results, we unearth several pivotal observations and insights fromextensive experiments: (1) When extending the rejection sampling strategy tothe multilingual context, it proves effective for model performances, albeitlimited. (2) Employing parallel corpora for math Supervised Fine-Tuning (SFT)across multiple languages not only significantly enhances model performancemultilingually but also elevates their monolingual performance. This indicatesthat crafting multilingual corpora can be regarded as a vital strategy forenhancing model performance in a specific language, especially in mathematicalreasoning tasks. For instance, MathOctopus-7B improves its counterparts thattrained on English from 42.2% to 50.8% on GSM8K testset.