Abstract
The pre-trained language models have achieved great successes in variousnatural language understanding (NLU) tasks due to its capacity to capture thedeep contextualized information in text by pre-training on large-scale corpora.One of the fundamental components in pre-trained language models is thevocabulary, especially for training multilingual models on many differentlanguages. In the technical report, we present our practices on trainingmultilingual pre-trained language models with BBPE: Byte-Level BPE (i.e., BytePair Encoding). In the experiment, we adopted the architecture of NEZHA as theunderlying pre-trained language model and the results show that NEZHA trainedwith byte-level subwords consistently outperforms Google multilingual BERT andvanilla NEZHA by a notable margin in several multilingual NLU tasks. We releasethe source code of our byte-level vocabulary building tools and themultilingual pre-trained language models.