Abstract
Recent advances in generative vision-language models (VLMs) have excitingpotential implications for AI in radiology, yet VLMs are also known to producehallucinations, nonsensical text, and other unwanted behaviors that can wasteclinicians' time and cause patient harm. Drawing on recent work on directpreference optimization (DPO), we propose a simple method for modifying thebehavior of pretrained VLMs performing radiology report generation bysuppressing unwanted types of generations. We apply our method to theprevention of hallucinations of prior exams, addressing a long-establishedproblem behavior in models performing chest X-ray report generation. Across ourexperiments, we find that DPO fine-tuning achieves a 3.2-4.8x reduction inlines hallucinating prior exams while maintaining model performance on clinicalaccuracy metrics. Our work is, to the best of our knowledge, the first work toapply DPO to medical VLMs, providing a data- and compute- efficient way tosuppress problem behaviors while maintaining overall clinical accuracy.