Abstract
We propose CounterCurate, a framework to comprehensively improve thevisio-linguistic compositional reasoning capability for both contrastive andgenerative multimodal models. In particular, we identify two criticalunder-explored problems: the neglect of the physically grounded reasoning(counting and position understanding) and the potential of using highly capabletext and image generation models for semantic counterfactual fine-tuning. Ourwork pioneers an approach that addresses these gaps. We first spotlight thenear-chance performance of multimodal models like CLIP and LLaVA in physicallygrounded compositional reasoning. We then apply simple data augmentation usinggrounded image generation model GLIGEN to generate fine-tuning data, resultingin significant performance improvements: +33% and +37% for CLIP and LLaVA,respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, weexploit the capabilities of high-performing text generation and imagegeneration models, specifically GPT-4V and DALLE-3, to curate challengingsemantic counterfactuals, thereby further enhancing compositional reasoningcapabilities on benchmarks such as SugarCrepe, where CounterCurate outperformsGPT-4V. To facilitate future research, we release our code, dataset, benchmark,and checkpoints at https://countercurate.github.io.