Abstract
This paper introduces TurnaboutLLM, a novel framework and dataset forevaluating the deductive reasoning abilities of Large Language Models (LLMs) byleveraging the interactive gameplay of detective games Ace Attorney andDanganronpa. The framework tasks LLMs with identifying contradictions betweentestimonies and evidences within long narrative contexts, a challenging taskdue to the large answer space and diverse reasoning types presented by itsquestions. We evaluate twelve state-of-the-art LLMs on the dataset, hinting atlimitations of popular strategies for enhancing deductive reasoning such asextensive thinking and Chain-of-Thought prompting. The results also suggestvarying effects of context size, the number of reasoning step and answer spacesize on model performance. Overall, TurnaboutLLM presents a substantialchallenge for LLMs' deductive reasoning abilities in complex, narrative-richenvironments.