Abstract
Contrastively-trained Vision-Language Models (VLMs), such as CLIP, havebecome the standard approach for learning discriminative vision-languagerepresentations. However, these models often exhibit shallow languageunderstanding, manifesting bag-of-words behaviour. These limitations arereinforced by their dual-encoder design, which induces a modality gap.Additionally, the reliance on vast web-collected data corpora for trainingmakes the process computationally expensive and introduces significant privacyconcerns. To address these limitations, in this work, we challenge thenecessity of vision encoders for retrieval tasks by introducing a vision-free,single-encoder retrieval pipeline. Departing from the traditional text-to-imageretrieval paradigm, we migrate to a text-to-text paradigm with the assistanceof VLLM-generated structured image descriptions. We demonstrate that thisparadigm shift has significant advantages, including a substantial reduction ofthe modality gap, improved compositionality, and better performance on shortand long caption queries, all attainable with only a few hours of calibrationon two GPUs. Additionally, substituting raw images with textual descriptionsintroduces a more privacy-friendly alternative for retrieval. To further assessgeneralisation and address some of the shortcomings of prior compositionalitybenchmarks, we release two benchmarks derived from Flickr30k and COCO,containing diverse compositional queries made of short captions, which we coinsubFlickr and subCOCO. Our vision-free retriever matches and often surpassestraditional multimodal models. Importantly, our approach achievesstate-of-the-art zero-shot performance on multiple retrieval andcompositionality benchmarks, with models as small as 0.3B parameters. Code isavailable at: https://github.com/IoannaNti/LexiCLIP