Learning Spoken Language Representations with Neural Lattice Language Modeling

  • 2020-07-06 10:38:03
  • Chao-Wei Huang, Yun-Nung Chen
  • 3

Abstract

Pre-trained language models have achieved huge improvement on many NLP tasks.However, these methods are usually designed for written text, so they do notconsider the properties of spoken language. Therefore, this paper aims atgeneralizing the idea of language model pre-training to lattices generated byrecognition systems. We propose a framework that trains neural lattice languagemodels to provide contextualized representations for spoken languageunderstanding tasks. The proposed two-stage pre-training approach reduces thedemands of speech data and has better efficiency. Experiments on intentdetection and dialogue act recognition datasets demonstrate that our proposedmethod consistently outperforms strong baselines when evaluated on spokeninputs. The code is available at https://github.com/MiuLab/Lattice-ELMo.

 

Quick Read (beta)

loading the full paper ...