Abstract

We present a memory-augmented approach to condition an autoregressivelanguage model on a knowledge graph. We represent the graph as a collection ofrelation triples and retrieve relevant relations for a given context to improvetext generation. Experiments on WikiText-103, WMT19, and enwik8 Englishdatasets demonstrate that our approach produces a better language model interms of perplexity and bits per character. We also show that relational memoryimproves coherence, is complementary to token-based memory, and enables causalinterventions. Our model provides a simple yet effective way to combine anautoregressive language model with a knowledge graph for a more coherent andlogical generation.

 

Quick Read (beta)