Abstract
Several recent papers have published good solutions for languageidentification (LID) for about 300 high-resource and medium-resource languages.However, there is no LID available that (i) covers a wide range of low-resourcelanguages, (ii) is rigorously evaluated and reliable and (iii) efficient andeasy to use. Here, we publish GlotLID-M, an LID model that satisfies thedesiderata of wide coverage, reliability and efficiency. It identifies 1665languages, a large increase in coverage compared to prior work. In ourexperiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID andNLLB) when balancing F1 and false positive rate (FPR). We analyze the uniquechallenges that low-resource LID poses: incorrect corpus metadata, leakage fromhigh-resource languages, difficulty separating closely related languages,handling of macrolanguage vs varieties and in general noisy data. We hope thatintegrating GlotLID-M into dataset creation pipelines will improve quality andenhance accessibility of NLP technology for low-resource languages andcultures. GlotLID-M model (including future versions), code, and list of datasources are available: https://github.com/cisnlp/GlotLID.