RoBERTuito: a pre-trained language model for social media text in Spanish

  • 2022-01-17 23:04:08
  • Juan Manuel Pérez, Damián A. Furman, Laura Alonso Alemany, Franco Luque
  • 0

Abstract

Since BERT appeared, Transformer language models and transfer learning havebecome state-of-the-art for Natural Language Understanding tasks. Recently,some works geared towards pre-training specially-crafted models for particulardomains, such as scientific papers, medical documents, user-generated texts,among others. These domain-specific models have been shown to improveperformance significantly in most tasks. However, for languages other thanEnglish such models are not widely available. In this work, we present RoBERTuito, a pre-trained language model foruser-generated text in Spanish, trained on over 500 million tweets. Experimentson a benchmark of tasks involving user-generated text showed that RoBERTuitooutperformed other pre-trained language models in Spanish. In addition to this,our model achieves top results for some English-Spanish tasks of the LinguisticCode-Switching Evaluation benchmark (LinCE) and has also competitiveperformance against monolingual models in English tasks. To facilitate furtherresearch, we make RoBERTuito publicly available at the HuggingFace model hubtogether with the dataset used to pre-train it.

 

Quick Read (beta)

loading the full paper ...