Abstract
Tokenization is a fundamental step in natural language processing, breakingtext into units that computational models can process. While learned subwordtokenizers have become the de-facto standard, they present challenges such aslarge vocabularies, limited adaptability to new domains or languages, andsensitivity to spelling errors and variations. To overcome these limitations,we investigate a hierarchical architecture for autoregressive languagemodelling that combines character-level and word-level processing. It employs alightweight character-level encoder to convert character sequences into wordembeddings, which are then processed by a word-level backbone model and decodedback into characters via a compact character-level decoder. This method retainsthe sequence compression benefits of word-level tokenization without relying ona rigid, predefined vocabulary. We demonstrate, at scales up to 7 billionparameters, that hierarchical transformers match the downstream taskperformance of subword-tokenizer-based models while exhibiting significantlygreater robustness to input perturbations. Additionally, during continuedpretraining on an out-of-domain language, our model trains almost twice asfast, achieves superior performance on the target language, and retains more ofits previously learned knowledge. Hierarchical transformers pave the way forNLP systems that are more robust, flexible, and generalizable across languagesand domains.