Abstract
Uniform downsampling remains the de facto standard for reducing spatialresolution in vision backbones. In this work, we propose an alternative designbuilt around a content-aware spatial grouping layer, that dynamically assignstokens to a reduced set based on image boundaries and their semantic content.Stacking our grouping layer across consecutive backbone stages results inhierarchical segmentation that arises natively in the feature extractionprocess, resulting in our coined Native Segmentation Vision Transformer. Weshow that a careful design of our architecture enables the emergence of strongsegmentation masks solely from grouping layers, that is, without additionalsegmentation-specific heads. This sets the foundation for a new paradigm ofnative, backbone-level segmentation, which enables strong zero-shot resultswithout mask supervision, as well as a minimal and efficient standalone modeldesign for downstream segmentation tasks. Our project page ishttps://research.nvidia.com/labs/dvl/projects/native-segmentation.