Abstract
Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC andConceptNet) poses unique challenges compared to the much studied conventionalknowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-formtext to represent nodes, resulting in orders of magnitude more nodes comparedto conventional KBs (18x more nodes in ATOMIC compared to Freebase(FB15K-237)). Importantly, this implies significantly sparser graph structures- a major challenge for existing KB completion methods that assume denselyconnected graphs over a relatively smaller set of nodes. In this paper, wepresent novel KB completion models that can address these challenges byexploiting the structural and semantic context of nodes. Specifically, weinvestigate two key ideas: (1) learning from local graph structure, using graphconvolutional networks and automatic graph densification and (2) transferlearning from pre-trained language models to knowledge graphs for enhancedcontextual representation of knowledge. We describe our method to incorporateinformation from both these sources in a joint model and provide the firstempirical results for KB completion on ATOMIC and evaluation with rankingmetrics on ConceptNet. Our results demonstrate the effectiveness of languagemodel representations in boosting link prediction performance and theadvantages of learning from local graph structure (+1.5 points in MRR forConceptNet) when training on subgraphs for computational efficiency. Furtheranalysis on model predictions shines light on the types of commonsenseknowledge that language models capture well.