Abstract
Vision-language models (VLMs) have achieved strong results on coding and mathbenchmarks that are challenging for humans, yet their ability to perform tasksthat come naturally to humans--such as perception, spatial navigation, andmemory management--remains understudied. Real video games are crafted to beintuitive for humans to learn and master by leveraging innate inductive biases,making them an ideal testbed for evaluating such capabilities in VLMs. To thisend, we introduce VideoGameBench, a benchmark consisting of 10 popular videogames from the 1990s that VLMs directly interact with in real-time.VideoGameBench challenges models to complete entire games with access to onlyraw visual inputs and a high-level description of objectives and controls, asignificant departure from existing setups that rely on game-specificscaffolding and auxiliary information. We keep three of the games secret toencourage solutions that generalize to unseen environments. Our experimentsshow that frontier vision-language models struggle to progress beyond thebeginning of each game. We find inference latency to be a major limitation offrontier models in the real-time setting; therefore, we introduceVideoGameBench Lite, a setting where the game pauses while waiting for the LM'snext action. The best performing model, Gemini 2.5 Pro, completes only 0.48% ofVideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalizationof the human skills mentioned above into this benchmark motivates progress inthese research directions.