Abstract
We interpret the function of individual neurons in CLIP by automaticallydescribing them using text. Analyzing the direct effects (i.e. the flow from aneuron through the residual stream to the output) or the indirect effects(overall contribution) fails to capture the neurons' function in CLIP.Therefore, we present the "second-order lens", analyzing the effect flowingfrom a neuron through the later attention heads, directly to the output. Wefind that these effects are highly selective: for each neuron, the effect issignificant for <2% of the images. Moreover, each effect can be approximated bya single direction in the text-image space of CLIP. We describe neurons bydecomposing these directions into sparse sets of text representations. The setsreveal polysemantic behavior - each neuron corresponds to multiple, oftenunrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, wemass-produce "semantic" adversarial examples by generating images with conceptsspuriously correlated to the incorrect class. Additionally, we use thesecond-order effects for zero-shot segmentation and attribute discovery inimages. Our results indicate that a scalable understanding of neurons can beused for model deception and for introducing new model capabilities.