Abstract
We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark forevaluation of instruction-following vision-language models for real-world use.Our starting point is curating 70 'instruction families' that we envisioninstruction tuned vision-language models should be able to address. Extendingbeyond evaluations like VQAv2 and COCO, tasks range from basic recognition togame playing and creative generation. Following curation, our dataset comprises592 test queries, each with a human-authored instruction-conditioned caption.These descriptions surface instruction-specific factors, e.g., for aninstruction asking about the accessibility of a storefront for wheelchairusers, the instruction-conditioned caption describes ramps/potential obstacles.These descriptions enable 1) collecting human-verified reference outputs foreach instance; and 2) automatic evaluation of candidate multimodal generationsusing a text-only LLM, aligning with human judgment. We quantify quality gapsbetween models and references using both human and automatic evaluations; e.g.,the top-performing instruction-following model wins against the GPT-4 referencein just 27% of the comparison. VisIT-Bench is dynamic to participate,practitioners simply submit their model's response on the project website;Data, code and leaderboard is available at visit-bench.github.io.