How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets

  • 2022-01-12 13:35:53
  • Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, Jörg Tiedemann
  • 2


A central question in natural language understanding (NLU) research iswhether high performance demonstrates the models' strong reasoningcapabilities. We present an extensive series of controlled experiments wherepre-trained language models are exposed to data that have undergone specificcorruption transformations. The transformations involve removing instances ofspecific word classes and often lead to non-sensical sentences. Our resultsshow that performance remains high for most GLUE tasks when the models arefine-tuned or tested on corrupted data, suggesting that the models leverageother cues for prediction even in non-sensical contexts. Our proposed datatransformations can be used as a diagnostic tool for assessing the extent towhich a specific dataset constitutes a proper testbed for evaluating models'language understanding capabilities.


Quick Read (beta)

loading the full paper ...