Abstract
Recent studies have shown that Large Vision-Language Models (VLMs) tend toneglect image content and over-rely on language-model priors, resulting inerrors in visually grounded tasks and hallucinations. We hypothesize that thisissue arises because existing VLMs are not explicitly trained to generate textsthat are accurately grounded in fine-grained image details. To enhance visualfeedback during VLM training, we propose S-VCO (Symmetrical Visual ContrastiveOptimization), a novel finetuning objective that steers the model towardcapturing important visual details and aligning them with corresponding texttokens. To further facilitate this detailed alignment, we introduce MVC, apaired image-text dataset built by automatically filtering and augmentingvisual counterfactual data to challenge the model with hard contrastive casesinvolving Minimal Visual Contrasts. Experiments show that our methodconsistently improves VLM performance across diverse benchmarks coveringvarious abilities and domains, achieving up to a 22% reduction inhallucinations, and significant gains in vision-centric and general tasks.Notably, these improvements become increasingly pronounced in benchmarks withhigher visual dependency. In short, S-VCO offers a significant enhancement ofVLM's visually-dependent task performance while retaining or even improving themodel's general abilities. We opensource our code at https://s-vco.github.io/