Neural Lattice Language Models

  • 2018-03-13 23:13:58
  • Jacob Buckman, Graham Neubig
  • 8

Abstract

In this work, we propose a new language modeling paradigm that has theability to perform both prediction and moderation of information flow atmultiple granularities: neural lattice language models. These models constructa lattice of possible paths through a sentence and marginalize across thislattice to calculate sequence probabilities or optimize parameters. Thisapproach allows us to seamlessly incorporate linguistic intuitions - includingpolysemy and existence of multi-word lexical items - into our language model.Experiments on multiple language modeling tasks show that English neurallattice language models that utilize polysemous embeddings are able to improveperplexity by 9.95% relative to a word-level baseline, and that a Chinese modelthat handles multi-character tokens is able to improve perplexity by 20.94%relative to a character-level baseline.

 

Quick Read (beta)

loading the full paper ...