Abstract
CLIP is one of the most popular foundation models and is heavily used formany vision-language tasks, yet little is known about its inner workings. AsCLIP is increasingly deployed in real-world applications, it is becoming evenmore critical to understand its limitations and embedded social biases tomitigate potentially harmful downstream consequences. However, the question ofwhat internal mechanisms drive both the impressive capabilities as well asproblematic shortcomings of CLIP has largely remained unanswered. To bridgethis gap, we study the conceptual consistency of text descriptions forattention heads in CLIP-like models. Specifically, we propose ConceptConsistency Score (CCS), a novel interpretability metric that measures howconsistently individual attention heads in CLIP models align with specificconcepts. Our soft-pruning experiments reveal that high CCS heads are criticalfor preserving model performance, as pruning them leads to a significantlylarger performance drop than pruning random or low CCS heads. Notably, we findthat high CCS heads capture essential concepts and play a key role inout-of-domain detection, concept-specific reasoning, and video-languageunderstanding. Moreover, we prove that high CCS heads learn spuriouscorrelations which amplify social biases. These results position CCS as apowerful interpretability metric exposing the paradox of performance and socialbiases in CLIP models.