Abstract
Vision-Language Models (VLMs) excel at visual understanding but often sufferfrom visual hallucinations, where they generate descriptions of nonexistentobjects, actions, or concepts, posing significant risks in safety-criticalapplications. Existing hallucination mitigation methods typically follow one oftwo paradigms: generation adjustment, which modifies decoding behavior to aligntext with visual inputs, and post-hoc verification, where external modelsassess and correct outputs. While effective, generation adjustment methodsoften rely on heuristics and lack correction mechanisms, while post-hocverification is complicated, typically requiring multiple models and tending toreject outputs rather than refine them. In this work, we introduce REVERSE, aunified framework that integrates hallucination-aware training with on-the-flyself-verification. By leveraging a new hallucination-verification datasetcontaining over 1.3M semi-synthetic samples, along with a novel inference-timeretrospective resampling technique, our approach enables VLMs to both detecthallucinations during generation and dynamically revise those hallucinations.Our evaluations show that REVERSE achieves state-of-the-art hallucinationreduction, outperforming the best existing methods by up to 12% on CHAIR-MSCOCOand 28% on HaloQuest. Our dataset, model, and code are available at:https://reverse-vlm.github.io.