Abstract

In this article, we explore the shallow heuristics used by transformer-basedpre-trained language models (PLMs) that are fine-tuned for natural languageinference (NLI). To do so, we construct or own dataset based on syllogistic,and we evaluate a number of models' performance on our dataset. We findevidence that the models rely heavily on certain shallow heuristics, picking upon symmetries and asymmetries between premise and hypothesis. We suggest thatthe lack of generalization observable in our study, which is becoming a topicof lively debate in the field, means that the PLMs are currently not learningNLI, but rather spurious heuristics.

 

Quick Read (beta)