SERENGETI: Massively Multilingual Language Models for Africa

  • 2023-05-26 21:20:04
  • Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Alcides Alcoba Inciarte
  • 0


Multilingual pretrained language models (mPLMs) acquire valuable,generalizable linguistic information during pretraining and have advanced thestate of the art on task-specific finetuning. To date, only ~31 out of ~2,000African languages are covered in existing language models. We ameliorate thislimitation by developing SERENGETI, a massively multilingual language modelthat covers 517 African languages and language varieties. We evaluate our novelmodels on eight natural language understanding tasks across 20 datasets,comparing to 4 mPLMs that cover 4-23 African languages. SERENGETI outperformsother models on 11 datasets across the eights tasks, achieving 82.27 averageF_1. We also perform analyses of errors from our models, which allows us toinvestigate the influence of language genealogy and linguistic similarity whenthe models are applied under zero-shot settings. We will publicly release ourmodels forresearch.\footnote{\href{}{}}


Quick Read (beta)

loading the full paper ...