Abstract
Most language understanding models in task-oriented dialog systems aretrained on a small amount of annotated training data, and evaluated in a smallset from the same distribution. However, these models can lead to systemfailure or undesirable output when being exposed to natural languageperturbation or variation in practice. In this paper, we conduct comprehensiveevaluation and analysis with respect to the robustness of natural languageunderstanding models, and introduce three important aspects related to languageunderstanding in real-world dialog systems, namely, language variety, speechcharacteristics, and noise perturbation. We propose a model-agnostic toolkitLAUG to approximate natural language perturbations for testing the robustnessissues in task-oriented dialog. Four data augmentation approaches covering thethree aspects are assembled in LAUG, which reveals critical robustness issuesin state-of-the-art models. The augmented dataset through LAUG can be used tofacilitate future research on the robustness testing of language understandingin task-oriented dialog.