Abstract
Traditional language models are unable to efficiently model entity namesobserved in text. All but the most popular named entities appear infrequentlyin text providing insufficient context. Recent efforts have recognized thatcontext can be generalized between entity names that share the same type (e.g.,\emph{person} or \emph{location}) and have equipped language models with accessto an external knowledge base (KB). Our Knowledge-Augmented Language Model(KALM) continues this line of work by augmenting a traditional model with a KB.Unlike previous methods, however, we train with an end-to-end predictiveobjective optimizing the perplexity of text. We do not require any additionalinformation such as named entity tags. In addition to improving languagemodeling performance, KALM learns to recognize named entities in an entirelyunsupervised way by using entity type information latent in the model. On aNamed Entity Recognition (NER) task, KALM achieves performance comparable withstate-of-the-art supervised models. Our work demonstrates that named entities(and possibly other types of world knowledge) can be modeled successfully usingpredictive learning and training on large corpora of text without anyadditional information.