Abstract
Unsupervised localization and segmentation are long-standing computer visionchallenges that involve decomposing an image into semantically-meaningfulsegments without any labeled data. These tasks are particularly interesting inan unsupervised setting due to the difficulty and cost of obtaining dense imageannotations, but existing unsupervised approaches struggle with complex scenescontaining multiple objects. Differently from existing methods, which arepurely based on deep learning, we take inspiration from traditional spectralsegmentation methods by reframing image decomposition as a graph partitioningproblem. Specifically, we examine the eigenvectors of the Laplacian of afeature affinity matrix from self-supervised networks. We find that theseeigenvectors already decompose an image into meaningful segments, and can bereadily used to localize objects in a scene. Furthermore, by clustering thefeatures associated with these segments across a dataset, we can obtainwell-delineated, nameable regions, i.e. semantic segmentations. Experiments oncomplex datasets (Pascal VOC, MS-COCO) demonstrate that our simple spectralmethod outperforms the state-of-the-art in unsupervised localization andsegmentation by a significant margin. Furthermore, our method can be readilyused for a variety of complex image editing tasks, such as background removaland compositing.