Abstract
Language models (LMs) have introduced a major paradigm shift in NaturalLanguage Processing (NLP) modeling where large pre-trained LMs became integralto most of the NLP tasks. The LMs are intelligent enough to find useful andrelevant representations of the language without any supervision. Perhaps,these models are used to fine-tune typical NLP tasks with significantly highaccuracy as compared to the traditional approaches. Conversely, the training ofthese models requires a massively large corpus that is a good representation ofthe language. English LMs generally perform better than their other languagecounterparts, due to the availability of massive English corpora. This work elaborates on the design and development of a large Arabic corpus.It consists of over 500 GB of Arabic cleaned text targeted at improvingcross-domain knowledge and downstream generalization capability of large-scalelanguage models. Moreover, the corpus is utilized in the training of a largeArabic LM. In order to evaluate the effectiveness of the LM, a number oftypical NLP tasks are fine-tuned. The tasks demonstrate a significant boostfrom 4.5 to 8.5% when compared to tasks fine-tuned on multi-lingual BERT(mBERT). To the best of my knowledge, this is currently the largest clean anddiverse Arabic corpus ever collected.