Abstract
Multimodal large language models (MLLMs) have achieved remarkable progress onvision-language tasks, yet their reasoning processes remain sometimesunreliable. We introduce PRISM-Bench, a benchmark of puzzle-based visualchallenges designed to evaluate not only whether models can solve problems, buthow their reasoning unfolds. Unlike prior evaluations that measure onlyfinal-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visualpuzzle and a step-by-step chain-of-thought (CoT) containing exactly one error,models must identify the first incorrect step. This setting enablesfine-grained assessment of logical consistency, error detection, and visualreasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric,and analogical reasoning, resisting shortcuts based on superficial patternmatching. Evaluations across state-of-the-art MLLMs reveal a persistent gapbetween fluent generation and faithful reasoning: models that produce plausibleCoTs often fail to locate simple logical faults. By disentangling answergeneration from reasoning verification, PRISM-Bench offers a sharper lens onmultimodal reasoning competence and underscores the need for diagnosticevaluation protocols in the development of trustworthy MLLMs.