Abstract
The advancement of Multimodal Large Language Models (MLLMs) has enabledsignificant progress in multimodal understanding, expanding their capacity toanalyze video content. However, existing evaluation benchmarks for MLLMsprimarily focus on abstract video comprehension, lacking a detailed assessmentof their ability to understand video compositions, the nuanced interpretationof how visual elements combine and interact within highly compiled videocontexts. We introduce VidComposition, a new benchmark specifically designed toevaluate the video composition understanding capabilities of MLLMs usingcarefully curated compiled videos and cinematic-level annotations.VidComposition includes 982 videos with 1706 multiple-choice questions,covering various compositional aspects such as camera movement, angle, shotsize, narrative structure, character actions and emotions, etc. Ourcomprehensive evaluation of 33 open-source and proprietary MLLMs reveals asignificant performance gap between human and model capabilities. Thishighlights the limitations of current MLLMs in understanding complex, compiledvideo compositions and offers insights into areas for further improvement. Theleaderboard and evaluation code are available athttps://yunlong10.github.io/VidComposition/.