Abstract

Low-resource languages such as Sinhala are often overlooked by open-sourceLarge Language Models (LLMs). In this research, we extend an existingmultilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLMtokenizer with Sinhala specific vocabulary and perform continual pre-trainingon a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. Thisis the very first decoder-based open-source LLM with explicit Sinhala support.When SinLlama was instruction fine-tuned for three text classification tasks,it outperformed base and instruct variants of Llama-3-8B by a significantmargin.

 

Quick Read (beta)