Abstract
How can we know whether new mechanistic interpretability methods achieve realimprovements? In pursuit of meaningful and lasting evaluation standards, wepropose MIB, a benchmark with two tracks spanning four tasks and five models.MIB favors methods that precisely and concisely recover relevant causalpathways or specific causal variables in neural language models. The circuitlocalization track compares methods that locate the model components - andconnections between them - most important for performing a task (e.g.,attribution patching or information flow routes). The causal variablelocalization track compares methods that featurize a hidden vector, e.g.,sparse autoencoders (SAEs) or distributed alignment search (DAS), and locatemodel features for a causal variable relevant to the task. Using MIB, we findthat attribution and mask optimization methods perform best on circuitlocalization. For causal variable localization, we find that the supervised DASmethod performs best, while SAE features are not better than neurons, i.e.,standard dimensions of hidden vectors. These findings illustrate that MIBenables meaningful comparisons of methods, and increases our confidence thatthere has been real progress in the field.