Abstract
Large Language models (LLMs) have demonstrated impressive performance on awide range of tasks, including in multimodal settings such as speech. However,their evaluation is often limited to English and a few high-resource languages.For low-resource languages, there is no standardized evaluation benchmark. Inthis paper, we address this gap by introducing mSTEB, a new benchmark toevaluate the performance of LLMs on a wide range of tasks covering languageidentification, text classification, question answering, and translation taskson both speech and text modalities. We evaluated the performance of leadingLLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art openmodels such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap inperformance between high-resource and low-resource languages, especially forlanguages spoken in Africa and Americas/Oceania. Our findings show that moreinvestment is needed to address their under-representation in LLMs coverage.