Abstract
User interaction with voice-powered agents generates large amounts ofunlabeled utterances. In this paper, we explore techniques to efficientlytransfer the knowledge from these unlabeled utterances to improve modelperformance on Spoken Language Understanding (SLU) tasks. We use Embeddingsfrom Language Model (ELMo) to take advantage of unlabeled data by learningcontextualized word representations. Additionally, we propose ELMo-Light(ELMoL), a faster and simpler unsupervised pre-training method for SLU. Ourfindings suggest unsupervised pre-training on a large corpora of unlabeledutterances leads to significantly better SLU performance compared to trainingfrom scratch and it can even outperform conventional supervised transfer.Additionally, we show that the gains from unsupervised transfer techniques canbe further improved by supervised transfer. The improvements are morepronounced in low resource settings and when using only 1000 labeled in-domainsamples, our techniques match the performance of training from scratch on10-15x more labeled in-domain data.