BERT (Bidirectional Encoder Representations from Transformers) and ALBERT (ALite BERT) are methods for pre-training language models which can later befine-tuned for a variety of Natural Language Understanding tasks. These methodshave been applied to a number of such tasks (mostly in English), achievingresults that outperform the state-of-the-art. In this paper, our contributionis twofold. First, we make available our trained BERT and Albert model forPortuguese. Second, we compare our monolingual and the standard multilingualmodels using experiments in semantic textual similarity, recognizing textualentailment, textual category classification, sentiment analysis, offensivecomment detection, and fake news detection, to assess the effectiveness of thegenerated language representations. The results suggest that both monolingualand multilingual models are able to achieve state-of-the-art and the advantageof training a single language model, if any, is small.