Abstract
Tokenizers act as a bridge between human language and the latent space oflanguage models, influencing how language is represented in these models. Dueto the immense popularity of English-Centric Large Language Models (LLMs),efforts are being made to adapt them for other languages. However, wedemonstrate that, from a tokenization standpoint, not all tokenizers offer fairrepresentation for complex script languages such as Tamil, Sinhala, and Hindi,primarily due to the choice of pre-tokenization methods. We go further to showthat pre-tokenization plays a more critical role than the tokenizationalgorithm itself in achieving an egalitarian representation of these complexscript languages. To address this, we introduce an improvement to the Byte PairEncoding (BPE) algorithm by incorporating graphemes, which we term GraphemePair Encoding (GPE). Our experiments show that grapheme-based characterextraction outperforms byte-level tokenizers for complex scripts. We validatethis approach through experiments on Tamil, Sinhala, and Hindi.