IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding

  • 2025-01-28 04:56:40
  • Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, Sreyoshi Bhaduri
  • 0

Abstract

Known by more than 1.5 billion people in the Indian subcontinent, Indiclanguages present unique challenges and opportunities for natural languageprocessing (NLP) research due to their rich cultural heritage, linguisticdiversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmarkdesigned to evaluate Large Language Models (LLMs) across Indic languages,building upon the MMLU Pro (Massive Multitask Language Understanding)framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the uniquechallenges and opportunities presented by the linguistic diversity of theIndian subcontinent. This benchmark encompasses a wide range of tasks inlanguage comprehension, reasoning, and generation, meticulously crafted tocapture the intricacies of Indian languages. IndicMMLU-Pro provides astandardized evaluation framework to push the research boundaries in Indiclanguage AI, facilitating the development of more accurate, efficient, andculturally sensitive models. This paper outlines the benchmarks' designprinciples, task taxonomy, and data collection methodology, and presentsbaseline results from state-of-the-art multilingual models.

 

Quick Read (beta)

loading the full paper ...