Abstract
Sketch segmentation involves grouping pixels within a sketch that belong tothe same object or instance. It serves as a valuable tool for sketch editingtasks, such as moving, scaling, or removing specific components. While imagesegmentation models have demonstrated remarkable capabilities in recent years,sketches present unique challenges for these models due to their sparse natureand wide variation in styles. We introduce SketchSeg, a method for instancesegmentation of raster scene sketches. Our approach adapts state-of-the-artimage segmentation and object detection models to the sketch domain byemploying class-agnostic fine-tuning and refining segmentation masks usingdepth cues. Furthermore, our method organizes sketches into sorted layers,where occluded instances are inpainted, enabling advanced sketch editingapplications. As existing datasets in this domain lack variation in sketchstyles, we construct a synthetic scene sketch segmentation dataset featuringsketches with diverse brush strokes and varying levels of detail. We use thisdataset to demonstrate the robustness of our approach and will release it topromote further research in the field. Project webpage: https://sketchseg.github.io/sketch-seg/