InkubaLM: A small language model for low-resource African languages

  • 2024-08-30 06:42:31
  • Atnafu Lambebo Tonja, Bonaventure F. P. Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, Benjamin Rosman
  • 0

Abstract

High-resource language models often fall short in the African context, wherethere is a critical need for models that are efficient, accessible, and locallyrelevant, even amidst significant computing and data constraints. This paperintroduces InkubaLM, a small language model with 0.4 billion parameters, whichachieves performance comparable to models with significantly larger parametercounts and more extensive training data on tasks such as machine translation,question-answering, AfriMMLU, and the AfriXnli task. Notably, InkubaLMoutperforms many larger models in sentiment analysis and demonstratesremarkable consistency across multiple languages. This work represents apivotal advancement in challenging the conventional paradigm that effectivelanguage models must rely on substantial resources. Our model and datasets arepublicly available \footnote{\url{https://huggingface.co/lelapa}} to encourageresearch and development on low-resource languages.

 

Quick Read (beta)

loading the full paper ...