Unified Language Model Pre-training for Natural Language Understanding and Generation

  • 2019-10-10 13:31:56
  • Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
  • 0

Abstract

This paper presents a new Unified pre-trained Language Model (UniLM) that canbe fine-tuned for both natural language understanding and generation tasks. Themodel is pre-trained using three types of language modeling objectives:unidirectional (both left-to-right and right-to-left), bidirectional, andsequence-to-sequence prediction. The unified modeling is achieved by employinga shared Transformer network and utilizing specific self-attention masks tocontrol what context the prediction conditions on. We can fine-tune UniLM as aunidirectional decoder, a bidirectional encoder, or a sequence-to-sequencemodel to support various downstream natural language understanding andgeneration tasks. UniLM compares favorably with BERT on the GLUE benchmark, andthe SQuAD 2.0 and CoQA question answering tasks. Moreover, our model achievesnew state-of-the-art results on three natural language generation tasks,including improving the CNN/DailyMail abstractive summarization ROUGE-L to40.63 (2.16 absolute improvement), pushing the CoQA generative questionanswering F1 score to 82.5 (37.1 absolute improvement), and the SQuAD questiongeneration BLEU-4 to 22.88 (6.50 absolute improvement). The code andpre-trained models are available at https://github.com/microsoft/unilm.

 

Quick Read (beta)

loading the full paper ...