Abstract
Pre-trained Language Models (PLMs) which are trained on large text corpus viaself-supervised learning method, have yielded promising performance on varioustasks in Natural Language Processing (NLP). However, though PLMs with hugeparameters can effectively possess rich knowledge learned from massive trainingtext and benefit downstream tasks at the fine-tuning stage, they still havesome limitations such as poor reasoning ability due to the lack of externalknowledge. Research has been dedicated to incorporating knowledge into PLMs totackle these issues. In this paper, we present a comprehensive review ofKnowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clearinsight into this thriving field. We introduce appropriate taxonomiesrespectively for Natural Language Understanding (NLU) and Natural LanguageGeneration (NLG) to highlight these two main tasks of NLP. For NLU, we dividethe types of knowledge into four categories: linguistic knowledge, textknowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG arecategorized into KG-based and retrieval-based methods. Finally, we point outsome promising future directions of KE-PLMs.