Abstract
We propose a permutation-based explanation method for image classifiers.Current image-model explanations like activation maps are limited toinstance-based explanations in the pixel space, making it difficult tounderstand global model behavior. In contrast, permutation based explanationsfor tabular data classifiers measure feature importance by comparing modelperformance on data before and after permuting a feature. We propose anexplanation method for image-based models that permutes interpretable conceptsacross dataset images. Given a dataset of images labeled with specific conceptslike captions, we permute a concept across examples in the text space and thengenerate images via a text-conditioned diffusion model. Feature importance isthen reflected by the change in model performance relative to unpermuted data.When applied to a set of concepts, the method generates a ranking of featureimportance. We show this approach recovers underlying model feature importanceon synthetic and real-world image classification tasks.