Abstract

Neural network models have shown great success at natural language inference(NLI), the task of determining whether a premise entails a hypothesis. However,recent studies suggest that these models may rely on fallible heuristics ratherthan deep language understanding. We introduce a challenge set to test whetherNLI systems adopt one such heuristic: assuming that a sentence entails all ofits subsequences, such as assuming that "Alice believes Mary is lying" entails"Alice believes Mary." We evaluate several competitive NLI models on thischallenge set and find strong evidence that they do rely on the subsequenceheuristic.

 

Quick Read (beta)