Pre-trained Models for Natural Language Processing: A Survey

  • 2020-03-18 15:22:51
  • Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang
  • 140

Abstract

Recently, the emergence of pre-trained models (PTMs) has brought naturallanguage processing (NLP) to a new era. In this survey, we provide acomprehensive review of PTMs for NLP. We first briefly introduce languagerepresentation learning and its research progress. Then we systematicallycategorize existing PTMs based on a taxonomy with four perspectives. Next, wedescribe how to adapt the knowledge of PTMs to the downstream tasks. Finally,we outline some potential directions of PTMs for future research. This surveyis purposed to be a hands-on guide for understanding, using, and developingPTMs for various NLP tasks.

 

Quick Read (beta)

loading the full paper ...