Abstract
Words in natural language follow a Zipfian distribution whereby some wordsare frequent but most are rare. Learning representations for words in the "longtail" of this distribution requires enormous amounts of data. Representationsof rare words trained directly on end tasks are usually poor, requiring us topre-train embeddings on external data, or treat all rare words asout-of-vocabulary words with a unique representation. We provide a method forpredicting embeddings of rare words on the fly from small amounts of auxiliarydata with a network trained end-to-end for the downstream task. We show thatthis improves results against baselines where embeddings are trained on the endtask for reading comprehension, recognizing textual entailment and languagemodeling.