Abstract
S\'ami, an indigenous language group comprising multiple languages, facesdigital marginalization due to the limited availability of data andsophisticated language models designed for its linguistic intricacies. Thiswork focuses on increasing technological participation for the S\'ami language.We draw the attention of the ML community towards the language modeling problemof Ultra Low Resource (ULR) languages. ULR languages are those for which theamount of available textual resources is very low, and the speaker count forthem is also very low. ULRLs are also not supported by mainstream LargeLanguage Models (LLMs) like ChatGPT, due to which gathering artificial trainingdata for them becomes even more challenging. Mainstream AI foundational modeldevelopment has given less attention to this category of languages. Generally,these languages have very few speakers, making it hard to find them. However,it is important to develop foundational models for these ULR languages topromote inclusion and the tangible abilities and impact of LLMs. To this end,we have compiled the available S\'ami language resources from the web to createa clean dataset for training language models. In order to study the behavior ofmodern LLM models with ULR languages (S\'ami), we have experimented withdifferent kinds of LLMs, mainly at the order of $\sim$ seven billionparameters. We have also explored the effect of multilingual LLM training forULRLs. We found that the decoder-only models under a sequential multilingualtraining scenario perform better than joint multilingual training, whereasmultilingual training with high semantic overlap, in general, performs betterthan training from scratch.This is the first study on the S\'ami language foradapting non-statistical language models that use the latest developments inthe field of natural language processing (NLP).