Abstract
Human intelligence requires correctness and robustness, with the former beingfoundational for the latter. In video understanding, correctness ensures theaccurate interpretation of visual content, and robustness maintains consistentperformance in challenging conditions. Despite advances in video large languagemodels (video LLMs), existing benchmarks inadequately reflect the gap betweenthese models and human intelligence in maintaining correctness and robustnessin video interpretation. We introduce the Video Thinking Test (Video-TT), toassess if video LLMs can interpret real-world videos as effectively as humans.Video-TT reflects genuine gaps in understanding complex visual narratives, andevaluates robustness against natural adversarial questions. Video-TT comprises1,000 YouTube Shorts videos, each with one open-ended question and fouradversarial questions that probe visual and narrative complexity. Ourevaluation shows a significant gap between video LLMs and human performance.