Abstract
This paper presents the Imputer, a neural sequence model that generatesoutput sequences iteratively via imputations. The Imputer is an iterativegenerative model, requiring only a constant number of generation stepsindependent of the number of input or output tokens. The Imputer can be trainedto approximately marginalize over all possible alignments between the input andoutput sequences, and all possible generation orders. We present a tractabledynamic programming training algorithm, which yields a lower bound on the logmarginal likelihood. When applied to end-to-end speech recognition, the Imputeroutperforms prior non-autoregressive models and achieves competitive results toautoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.