Abstract
Large Vision-Language Models (LVLMs) are capable of handling diverse datatypes such as imaging, text, and physiological signals, and can be applied invarious fields. In the medical field, LVLMs have a high potential to offersubstantial assistance for diagnosis and treatment. Before that, it is crucialto develop benchmarks to evaluate LVLMs' effectiveness in various medicalapplications. Current benchmarks are often built upon specific academicliterature, mainly focusing on a single domain, and lacking varying perceptualgranularities. Thus, they face specific challenges, including limited clinicalrelevance, incomplete evaluations, and insufficient guidance for interactiveLVLMs. To address these limitations, we developed the GMAI-MMBench, the mostcomprehensive general medical AI benchmark with well-categorized data structureand multi-perceptual granularity to date. It is constructed from 284 datasetsacross 38 medical image modalities, 18 clinical-related tasks, 18 departments,and 4 perceptual granularities in a Visual Question Answering (VQA) format.Additionally, we implemented a lexical tree structure that allows users tocustomize evaluation tasks, accommodating various assessment needs andsubstantially supporting medical AI research and applications. We evaluated 50LVLMs, and the results show that even the advanced GPT-4o only achieves anaccuracy of 53.96%, indicating significant room for improvement. Moreover, weidentified five key insufficiencies in current cutting-edge LVLMs that need tobe addressed to advance the development of better medical applications. Webelieve that GMAI-MMBench will stimulate the community to build the nextgeneration of LVLMs toward GMAI.