Abstract
The multilingual BERT model is trained on 104 languages and meant to serve asa universal language model and tool for encoding sentences. We explore how wellthe model performs on several languages across several tasks: a diagnosticclassification probing the embeddings for a particular syntactic property, acloze task testing the language modelling ability to fill in gaps in asentence, and a natural language generation task testing for the ability toproduce coherent text fitting a given context. We find that the currentlyavailable multilingual BERT model is clearly inferior to the monolingualcounterparts, and cannot in many cases serve as a substitute for a well-trainedmonolingual model. We find that the English and German models perform well atgeneration, whereas the multilingual model is lacking, in particular, forNordic languages.