Abstract
We study the intriguing connection between visual data, deep networks, andthe brain. Our method creates a universal channel alignment by using brainvoxel fMRI response prediction as the training objective. We discover that deepnetworks, trained with different objectives, share common feature channelsacross various models. These channels can be clustered into recurring sets,corresponding to distinct brain regions, indicating the formation of visualconcepts. Tracing the clusters of channel responses onto the images, we seesemantically meaningful object segments emerge, even without any superviseddecoder. Furthermore, the universal feature alignment and the clustering ofchannels produce a picture and quantification of how visual information isprocessed through the different network layers, which produces precisecomparisons between the networks.