Types of Out-of-Distribution Texts and How to Detect Them

  • 2021-09-14 17:12:38
  • Udit Arora, William Huang, He He
  • 0

Abstract

Despite agreement on the importance of detecting out-of-distribution (OOD)examples, there is little consensus on the formal definition of OOD examplesand how to best detect them. We categorize these examples by whether theyexhibit a background shift or a semantic shift, and find that the two majorapproaches to OOD detection, model calibration and density estimation (languagemodeling for text), have distinct behavior on these types of OOD data. Across14 pairs of in-distribution and OOD English natural language understandingdatasets, we find that density estimation methods consistently beat calibrationmethods in background shift settings, while performing worse in semantic shiftsettings. In addition, we find that both methods generally fail to detectexamples from challenge data, highlighting a weak spot for current methods.Since no single method works well across all settings, our results call for anexplicit definition of OOD examples when evaluating different detectionmethods.

 

Quick Read (beta)

loading the full paper ...