Abstract
Generating detailed and accurate descriptions for specific regions in imagesand videos remains a fundamental challenge for vision-language models. Weintroduce the Describe Anything Model (DAM), a model designed for detailedlocalized captioning (DLC). DAM preserves both local details and global contextthrough two key innovations: a focal prompt, which ensures high-resolutionencoding of targeted regions, and a localized vision backbone, which integratesprecise localization with its broader context. To tackle the scarcity ofhigh-quality DLC data, we propose a Semi-supervised learning (SSL)-based DataPipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets andexpands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmarkdesigned to evaluate DLC without relying on reference captions. DAM sets newstate-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, anddetailed multi-sentence localized image and video captioning.