Abstract
Language models perform differently across languages. It has been previouslysuggested that morphological typology may explain some of this variability(Cotterell et al., 2018). We replicate previous analyses and find additionalnew evidence for a performance gap between agglutinative and fusionallanguages, where fusional languages, such as English, tend to have betterlanguage modeling performance than morphologically more complex languages likeTurkish. We then propose and test three possible causes for this performancegap: morphological alignment of tokenizers, tokenization quality, anddisparities in dataset sizes and measurement. To test the morphologicalalignment hypothesis, we present MorphScore, a tokenizer evaluation metric, andsupporting datasets for 22 languages. We find some evidence that tokenizationquality explains the performance gap, but none for the role of morphologicalalignment. Instead we find that the performance gap is most reduced whentraining datasets are of equivalent size across language types, but only whenscaled according to the so-called "byte-premium" -- the different encodingefficiencies of different languages and orthographies. These results suggestthat no language is harder or easier for a language model to learn on the basisof its morphological typology. Differences in performance can be attributed todisparities in dataset size. These results bear on ongoing efforts to improveperformance for low-performing and under-resourced languages.