Abstract
Query rewriting (QR) is an increasingly important technique to reducecustomer friction caused by errors in a spoken language understanding pipeline,where the errors originate from various sources such as speech recognitionerrors, language understanding errors or entity resolution errors. In thiswork, we first propose a neural-retrieval based approach for query rewriting.Then, inspired by the wide success of pre-trained contextual languageembeddings, and also as a way to compensate for insufficient QR training data,we propose a language-modeling (LM) based approach to pre-train queryembeddings on historical user conversation data with a voice assistant. Inaddition, we propose to use the NLU hypotheses generated by the languageunderstanding system to augment the pre-training. Our experiments showpre-training provides rich prior information and help the QR task achievestrong performance. We also show joint pre-training with NLU hypotheses hasfurther benefit. Finally, after pre-training, we find a small set of rewritepairs is enough to fine-tune the QR model to outperform a strong baseline byfull training on all QR training data.