Abstract
The ability to organically reason over and with both text and images is apillar of human intelligence, yet the ability of Multimodal Large LanguageModels (MLLMs) to perform such multimodal reasoning remains under-explored.Existing benchmarks often emphasize text-dominant reasoning or rely on shallowvisual cues, failing to adequately assess integrated visual and textualreasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmarktargeting organic multimodal reasoning across mathematics, physics, chemistry,and coding. EMMA tasks demand advanced cross-modal reasoning that cannot beaddressed by reasoning independently in each modality, offering an enhancedtest suite for MLLMs' reasoning capabilities. Our evaluation ofstate-of-the-art MLLMs on EMMA reveals significant limitations in handlingcomplex multimodal and multi-step reasoning tasks, even with advancedtechniques like Chain-of-Thought prompting and test-time compute scalingunderperforming. These findings underscore the need for improved multimodalarchitectures and training paradigms to close the gap between human and modelreasoning in multimodality.