Abstract
Reasoning is a key ability for an intelligent system. Large language models(LMs) achieve above-chance performance on abstract reasoning tasks, but exhibitmany imperfections. However, human abstract reasoning is also imperfect. Forexample, human reasoning is affected by our real-world knowledge and beliefs,and shows notable "content effects"; humans reason more reliably when thesemantic content of a problem supports the correct logical inferences. Thesecontent-entangled reasoning patterns play a central role in debates about thefundamental nature of human intelligence. Here, we investigate whether languagemodels $\unicode{x2014}$ whose prior expectations capture some aspects of humanknowledge $\unicode{x2014}$ similarly mix content into their answers to logicalproblems. We explored this question across three logical reasoning tasks:natural language inference, judging the logical validity of syllogisms, and theWason selection task. We evaluate state of the art large language models, aswell as humans, and find that the language models reflect many of the samepatterns observed in humans across these tasks $\unicode{x2014}$ like humans,models answer more accurately when the semantic content of a task supports thelogical inferences. These parallels are reflected both in answer patterns, andin lower-level features like the relationship between model answerdistributions and human response times. Our findings have implications forunderstanding both these cognitive effects in humans, and the factors thatcontribute to language model performance.