Abstract
Developments in machine learning interpretability techniques over the pastdecade have provided new tools to observe the image regions that are mostinformative for classification and localization in artificial neural networks(ANNs). Are the same regions similarly informative to human observers? Usingdata from 78 new experiments and 6,610 participants, we show that passiveattention techniques reveal a significant overlap with human visual selectivityestimates derived from 6 distinct behavioral tasks including visualdiscrimination, spatial localization, recognizability, free-viewing,cued-object search, and saliency search fixations. We find that inputvisualizations derived from relatively simple ANN architectures probed usingguided backpropagation methods are the best predictors of a shared component inthe joint variability of the human measures. We validate these correlationalresults with causal manipulations using recognition experiments. We show thatimages masked with ANN attention maps were easier for humans to classify thancontrol masks in a speeded recognition experiment. Similarly, we find thatrecognition performance in the same ANN models was likewise influenced bymasking input images using human visual selectivity maps. This work contributesa new approach to evaluating the biological and psychological validity ofleading ANNs as models of human vision: by examining their similarities anddifferences in terms of their visual selectivity to the information containedin images.