Abstract
We present an approach for building an active agent that learns to segmentits visual observations into individual objects by interacting with itsenvironment in a completely self-supervised manner. The agent uses its currentsegmentation model to infer pixels that constitute objects and refines thesegmentation model by interacting with these pixels. The model learned fromover 50K interactions generalizes to novel objects and backgrounds. To dealwith noisy training signal for segmenting objects obtained by self-supervisedinteractions, we propose robust set loss. A dataset of robot's interactionsalong-with a few human labeled examples is provided as a benchmark for futureresearch. We test the utility of the learned segmentation model by providingresults on a downstream vision-based control task of rearranging multipleobjects into target configurations from visual inputs alone. Videos, code, androbotic interaction dataset are available athttps://pathak22.github.io/seg-by-interaction/