CoCoLM: COmplex COmmonsense Enhanced Language Model

  • 2020-12-31 15:05:36
  • Changlong Yu, Hongming Zhang, Yangqiu Song, Wilfred Ng
  • 0

Abstract

Large-scale pre-trained language models have demonstrated strong knowledgerepresentation ability. However, recent studies suggest that even though thesegiant models contains rich simple commonsense knowledge (e.g., bird can fly andfish can swim.), they often struggle with the complex commonsense knowledgethat involves multiple eventualities (verb-centric phrases, e.g., identifyingthe relationship between ``Jim yells at Bob'' and ``Bob is upset'').To addressthis problem, in this paper, we propose to help pre-trained language modelsbetter incorporate complex commonsense knowledge. Different from existingfine-tuning approaches, we do not focus on a specific task and propose ageneral language model named CoCoLM. Through the careful training over alarge-scale eventuality knowledge graphs ASER, we successfully teachpre-trained language models (i.e., BERT and RoBERTa) rich complex commonsenseknowledge among eventualities. Experiments on multiple downstream commonsensetasks that requires the correct understanding of eventualities demonstrate theeffectiveness of CoCoLM.

 

Quick Read (beta)

loading the full paper ...