Abstract
We show how the spellings of known words can help us deal with unknown wordsin open-vocabulary NLP tasks. The method we propose can be used to extend anyclosed-vocabulary generative model, but in this paper we specifically considerthe case of neural language modeling. Our Bayesian generative story combines astandard RNN language model (generating the word tokens in each sentence) withan RNN-based spelling model (generating the letters in each word type). Thesetwo RNNs respectively capture sentence structure and word structure, and arekept separate as in linguistics. By invoking the second RNN to generatespellings for novel words in context, we obtain an open-vocabulary languagemodel. For known words, embeddings are naturally inferred by combining evidencefrom type spelling and token context. Comparing to baselines (including a novelstrong baseline), we beat previous work and establish state-of-the-art resultson multiple datasets.