Abstract
Large language models (LLMs) need to serve everyone, including a globalmajority of non-English speakers. However, most LLMs today, and open LLMs inparticular, are often intended for use in just English (e.g. Llama2, Mistral)or a small handful of high-resource languages (e.g. Mixtral, Qwen). Recentresearch shows that, despite limits in their intended use, people prompt LLMsin many different languages. Therefore, in this paper, we investigate the basicmultilingual capabilities of state-of-the-art open LLMs beyond their intendeduse. For this purpose, we introduce MultiQ, a new silver standard benchmark forbasic open-ended question answering with 27.4k test questions across atypologically diverse set of 137 languages. With MultiQ, we evaluate languagefidelity, i.e. whether models respond in the prompted language, and questionanswering accuracy. All LLMs we test respond faithfully and/or accurately forat least some languages beyond their intended use. Most models are moreaccurate when they respond faithfully. However, differences across models arelarge, and there is a long tail of languages where models are neither accuratenor faithful. We explore differences in tokenization as a potential explanationfor our findings, identifying possible correlations that warrant furtherinvestigation.