LaoPLM: Pre-trained Language Models for Lao

  • 2021-10-14 09:01:34
  • Nankai Lin, Yingwen Fu, Chuwei Chen, Ziyu Yang, Shengyi Jiang
  • 0

Abstract

Trained on the large corpus, pre-trained language models (PLMs) can capturedifferent levels of concepts in context and hence generate universal languagerepresentations. They can benefit multiple downstream natural languageprocessing (NLP) tasks. Although PTMs have been widely used in most NLPapplications, especially for high-resource languages such as English, it isunder-represented in Lao NLP research. Previous work on Lao has been hamperedby the lack of annotated datasets and the sparsity of language resources. Inthis work, we construct a text classification dataset to alleviate theresource-scare situation of the Lao language. We additionally present the firsttransformer-based PTMs for Lao with four versions: BERT-small, BERT-base,ELECTRA-small and ELECTRA-base, and evaluate it over two downstream tasks:part-of-speech tagging and text classification. Experiments demonstrate theeffectiveness of our Lao models. We will release our models and datasets to thecommunity, hoping to facilitate the future development of Lao NLP applications.

 

Quick Read (beta)

loading the full paper ...