Abstract
Massive language models are the core of modern NLP modeling and have beenshown to encode impressive amounts of commonsense and factual information.However, that knowledge exists only within the latent parameters of the model,inaccessible to inspection and interpretation, and even worse, factualinformation memorized from the training corpora is likely to become stale asthe world changes. Knowledge stored as parameters will also inevitably exhibitall of the biases inherent in the source materials. To address these problems,we develop a neural language model that includes an explicit interface betweensymbolically interpretable factual information and subsymbolic neuralknowledge. We show that this model dramatically improves performance on twoknowledge-intensive question-answering tasks. More interestingly, the model canbe updated without re-training by manipulating its symbolic representations. Inparticular this model allows us to add new facts and overwrite existing ones inways that are not possible for earlier models.