Abstract
We propose and study a novel 'Panoptic Segmentation' (PS) task. Panopticsegmentation unifies the traditionally distinct tasks of instance segmentation(detect and segment each object instance) and semantic segmentation (assign aclass label to each pixel). The unification is natural and presents novelalgorithmic challenges not present in either instance or semantic segmentationwhen studied in isolation. To measure performance on the task, we introduce apanoptic quality (PQ) measure, and show that it is simple and interpretable.Using PQ, we study human performance on three existing datasets that have thenecessary annotations for PS, which helps us better understand the task andmetric. We also propose a basic algorithmic approach to combine instance andsemantic segmentation outputs into panoptic outputs and compare this to humanperformance. PS can serve as foundation of future challenges in segmentationand visual recognition. Our goal is to drive research in novel directions byinviting the community to explore the proposed panoptic segmentation task.