Abstract
English pretrained language models, which make up the backbone of many modernNLP systems, require huge amounts of unlabeled training data. These models aregenerally presented as being trained only on English text but have been foundto transfer surprisingly well to other languages. We investigate thisphenomenon and find that common English pretraining corpora actually containsignificant amounts of non-English text: even when less than 1% of data is notEnglish (well within the error rate of strong language classifiers), this leadsto hundreds of millions of foreign language tokens in large-scale datasets. Wethen demonstrate that even these small percentages of non-English datafacilitate cross-lingual transfer for models trained on them, with targetlanguage performance strongly correlated to the amount of in-language data seenduring pretraining. In light of these findings, we argue that no model is trulymonolingual when pretrained at scale, which should be considered whenevaluating cross-lingual transfer.