Abstract
Attention-based recurrent neural encoder-decoder models present an elegantsolution to the automatic speech recognition problem. This approach folds theacoustic model, pronunciation model, and language model into a single networkand requires only a parallel corpus of speech and text for training. However,unlike in conventional approaches that combine separate acoustic and languagemodels, it is not clear how to use additional (unpaired) text. While there hasbeen previous work on methods addressing this problem, a thorough comparisonamong methods is still lacking. In this paper, we compare a suite of pastmethods and some of our own proposed methods for using unpaired text data toimprove encoder-decoder models. For evaluation, we use the medium-sizedSwitchboard data set and the large-scale Google voice search and dictation datasets. Our results confirm the benefits of using unpaired text across a range ofmethods and data sets. Surprisingly, for first-pass decoding, the rather simpleapproach of shallow fusion performs best across data sets. However, for Googledata sets we find that cold fusion has a lower oracle error rate andoutperforms other approaches after second-pass rescoring on the Google voicesearch data set.