Abstract
Accurate detection of offensive language is essential for a number ofapplications related to social media safety. There is a sharp contrast inperformance in this task between low and high-resource languages. In thispaper, we adapt fine-tuning strategies that have not been previously exploredfor Sinhala in the downstream task of offensive language detection. Using thisapproach, we introduce four models: "Subasa-XLM-R", which incorporates anintermediate Pre-Finetuning step using Masked Rationale Prediction. Twovariants of "Subasa-Llama" and "Subasa-Mistral", are fine-tuned versions ofLlama (3.2) and Mistral (v0.3), respectively, with a task-specific strategy. Weevaluate our models on the SOLD benchmark dataset for Sinhala offensivelanguage detection. All our models outperform existing baselines. Subasa-XLM-Rachieves the highest Macro F1 score (0.84) surpassing state-of-the-art largelanguage models like GPT-4o when evaluated on the same SOLD benchmark datasetunder zero-shot settings. The models and code are publicly available.