Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation

  • 2020-09-10 16:46:46
  • Junlong Li, Zhuosheng Zhang, Hai Zhao, Xi Zhou, Xiang Zhou
  • 7


Pre-trained Language Models (PrLMs) have been widely used as backbones inlots of Natural Language Processing (NLP) tasks. The common process ofutilizing PrLMs is first pre-training on large-scale general corpora withtask-independent LM training objectives, then fine-tuning on task datasets withtask-specific training objectives. Pre-training in a task-independent wayenables the models to learn language representations, which is universal tosome extent, but fails to capture crucial task-specific features in themeantime. This will lead to an incompatibility between pre-training andfine-tuning. To address this issue, we introduce task-specific pre-training onin-domain task-related corpora with task-specific objectives. This procedure isplaced between the original two stages to enhance the model understandingcapacity of specific tasks. In this work, we focus on Dialogue-related NaturalLanguage Processing (DrNLP) tasks and design a Dialogue-Adaptive Pre-trainingObjective (DAPO) based on some important qualities for assessing dialogueswhich are usually ignored by general LM pre-training objectives. PrLMs withDAPO on a large in-domain dialogue corpus are then fine-tuned for downstreamDrNLP tasks. Experimental results show that models with DAPO surpass those withgeneral LM pre-training objectives and other strong baselines on downstreamDrNLP tasks.


Quick Read (beta)

loading the full paper ...