Abstract
Detecting disinformation that blends manipulated text and images has becomeincreasingly challenging, as AI tools make synthetic content easy to generateand disseminate. While most existing AI safety benchmarks focus on singlemodality misinformation (i.e., false content shared without intent to deceive),intentional multimodal disinformation, such as propaganda or conspiracytheories that imitate credible news, remains largely unaddressed. We introducethe Vision-Language Disinformation Detection Benchmark (VLDBench), the firstlarge-scale resource supporting both unimodal (text-only) and multimodal (text+ image) disinformation detection. VLDBench comprises approximately 62,000labeled text-image pairs across 13 categories, curated from 58 news outlets.Using a semi-automated pipeline followed by expert review, 22 domain expertsinvested over 500 hours to produce high-quality annotations with substantialinter-annotator agreement. Evaluations of state-of-the-art Large LanguageModels (LLMs) and Vision-Language Models (VLMs) on VLDBench show thatincorporating visual cues improves detection accuracy by 5 to 35 percentagepoints over text-only models. VLDBench provides data and code for evaluation,fine-tuning, and robustness testing to support disinformation analysis.Developed in alignment with AI governance frameworks (e.g., the MIT AI RiskRepository), VLDBench offers a principled foundation for advancing trustworthydisinformation detection in multimodal media. Project: https://vectorinstitute.github.io/VLDBench/ Dataset:https://huggingface.co/datasets/vector-institute/VLDBench Code:https://github.com/VectorInstitute/VLDBench