Abstract
Language models (LMs) have introduced a major paradigm shift in NaturalLanguage Processing (NLP) modeling where large pre-trained LMs became integralto most of the NLP tasks. The LMs are intelligent enough to find useful andrelevant representations of the language without any supervision. Perhaps,these models are used to fine-tune typical NLP tasks with significantly highaccuracy as compared to the traditional approaches. Conversely, the training ofthese models requires a massively large corpus that is a good representation ofthe language. English LMs generally perform better than their other languagecounterparts, due to the availability of massive English corpora. This workelaborates on the design and development of a large Arabic corpus. It consistsof over 500 GB of Arabic cleaned text targeted at improving cross-domainknowledge and downstream generalization capability of large-scale languagemodels. Moreover, the corpus is utilized in the training of a large Arabic LM.In order to evaluate the effectiveness of the LM, a number of typical NLP tasksare fine-tuned. The tasks demonstrate a significant boost from 4.5 to 8.5% whencompared to tasks fine-tuned on multi-lingual BERT (mBERT). To the best of myknowledge, this is currently the largest clean and diverse Arabic corpus evercollected.