Abstract
Byte-level language models eliminate fragile tokenizers but facecomputational challenges in morphologically-rich languages (MRLs), where wordsspan many bytes. We propose H-NET++, a hierarchical dynamic-chunking model thatlearns linguistically-informed segmentation through end-to-end training. Keyinnovations include: (1) a lightweight Transformer context-mixer (1.9Mparameters) for cross-chunk attention, (2) a two-level latent hyper-prior fordocument-level consistency, (3) specialized handling of orthographic artifacts(e.g. Persian ZWNJ), and (4) curriculum-based training with staged sequencelengths. On a 1.4B-token Persian corpus, H-NET++ achieves state-of-the-artresults: 0.159 BPB reduction versus BPE-based GPT-2-fa (12% bettercompression), 5.4pp gain on ParsGLUE, 53% improved robustness to ZWNJcorruption, and 73.8% F1 on gold morphological boundaries. Our learned chunksalign with Persian morphology without explicit supervision, demonstrating thathierarchical dynamic chunking provides an effective tokenizer-free solution forMRLs while maintaining computational efficiency.