Abstract
Existing pre-trained language models (PLMs) have demonstrated theeffectiveness of self-supervised learning for a broad range of natural languageprocessing (NLP) tasks. However, most of them are not explicitly aware ofdomain-specific knowledge, which is essential for downstream tasks in manydomains, such as tasks in e-commerce scenarios. In this paper, we proposeK-PLUG, a knowledge-injected pre-trained language model based on theencoder-decoder transformer that can be transferred to both natural languageunderstanding and generation tasks. We verify our method in a diverse range ofe-commerce scenarios that require domain-specific knowledge. Specifically, wepropose five knowledge-aware self-supervised pre-training objectives toformulate the learning of domain-specific knowledge, including e-commercedomain-specific knowledge-bases, aspects of product entities, categories ofproduct entities, and unique selling propositions of product entities. K-PLUGachieves new state-of-the-art results on a suite of domain-specific NLP tasks,including product knowledge base completion, abstractive product summarization,and multi-turn dialogue, significantly outperforms baselines across the board,which demonstrates that the proposed method effectively learns a diverse set ofdomain-specific knowledge for both language understanding and generation tasks.