Abstract
Large language models with vision capabilities (VLMs), e.g., GPT-4o andGemini 1.5 Pro are powering countless image-text applications and scoring highon many vision-understanding benchmarks. We propose BlindTest, a suite of 7visual tasks absurdly easy to humans such as identifying (a) whether twocircles overlap; (b) whether two lines intersect; (c) which letter is beingcircled in a word; and (d) counting the number of circles in a Olympic-likelogo. Surprisingly, four state-of-the-art VLMs are, on average, only 56.20%accurate on our benchmark, with \newsonnet being the best (73.77% accuracy). OnBlindTest, VLMs struggle with tasks that requires precise spatial informationand counting (from 0 to 10), sometimes providing an impression of a person withmyopia seeing fine details as blurry and making educated guesses. Code isavailable at: https://vlmsareblind.github.io/