Abstract
Instance segmentation of novel objects instances in RGB images, given someexample images for each object, is a well known problem in computer vision.Designing a model general enough to be employed for all kinds of novel objectswithout (re-) training has proven to be a difficult task. To handle this, wepresent a new training-free framework, called: Novel Object Cyclic Thresholdbased Instance Segmentation (NOCTIS). NOCTIS integrates two pre-trained models:Grounded-SAM 2 for object proposals with precise bounding boxes andcorresponding segmentation masks; and DINOv2 for robust class and patchembeddings, due to its zero-shot capabilities. Internally, the proposal-objectmatching is realized by determining an object matching score based on thesimilarity of the class embeddings and the average maximum similarity of thepatch embeddings with a new cyclic thresholding (CT) mechanism that mitigatesunstable matches caused by repetitive textures or visually similar patterns.Beyond CT, NOCTIS introduces: (i) an appearance score that is unaffected byobject selection bias; (ii) the usage of the average confidence of theproposals bounding box and mask as a scoring component; and (iii) an RGB-onlypipeline that performs even better than RGB-D ones. We empirically show thatNOCTIS, without further training/fine tuning, attains state-of-the-art resultsregarding the mean AP score, w.r.t. the best RGB and RGB-D methods on the sevencore datasets of the BOP 2023 challenge for the "Model-based 2D segmentation ofunseen objects" task.