Abstract
Large vision-language models (LVLMs) achieve impressive performance onmultimodal tasks but often suffer from hallucination, and confidently describeobjects or attributes not present in the image. Current inference-timeinterventions, while training-free, struggle to maintain accuracy in open-endedand long-form generation scenarios. We introduce the Confidence-Aware AttentionCalibration (CAAC) framework to address this challenge by targeting two keybiases: spatial perception bias, which distributes attention disproportionatelyacross image tokens, and modality bias, which shifts focus from visual totextual inputs over time. CAAC employs a two-step approach: Visual-TokenCalibration (VTC) to balance attention across visual tokens, and AdaptiveAttention Re-Scaling (AAR) to reinforce visual grounding based on the model'sconfidence. This confidence-driven adjustment ensures consistent visualalignment during generation. Experiments on CHAIR, AMBER, and POPE benchmarksdemonstrate that CAAC outperforms baselines, particularly in long-formgenerations, effectively reducing hallucination.