Abstract
Generative AI models have impressive performance on many Natural LanguageProcessing tasks such as language understanding, reasoning and languagegeneration. One of the most important questions that is being asked by the AIcommunity today is about the capabilities and limits of these models, and it isclear that evaluating generative AI is very challenging. Most studies ongenerative Large Language Models (LLMs) are restricted to English and it isunclear how capable these models are at understanding and generating otherlanguages. We present the first comprehensive benchmarking of generative LLMs -MEGA, which evaluates models on standard NLP benchmarks, covering 8 diversetasks and 33 typologically diverse languages. We also compare the performanceof generative LLMs to State of the Art (SOTA) non-autoregressive models onthese tasks to determine how well generative models perform compared to theprevious generation of LLMs. We present a thorough analysis of the performanceof models across languages and discuss some of the reasons why generative LLMsare currently not optimal for all languages. We create a framework forevaluating generative LLMs in the multilingual setting and provide directionsfor future progress in the field.