Abstract
We propose UniSeg3D, a unified 3D segmentation framework that achievespanoptic, semantic, instance, interactive, referring, and open-vocabularysemantic segmentation tasks within a single model. Most previous 3Dsegmentation approaches are specialized for a specific task, thereby limitingtheir understanding of 3D scenes to a task-specific perspective. In contrast,the proposed method unifies six tasks into unified representations processed bythe same Transformer. It facilitates inter-task knowledge sharing and,therefore, promotes comprehensive 3D scene understanding. To take advantage ofmulti-task unification, we enhance the performance by leveraging taskconnections. Specifically, we design a knowledge distillation method and acontrastive learning method to transfer task-specific knowledge acrossdifferent tasks. Benefiting from extensive inter-task knowledge sharing, ourUniSeg3D becomes more powerful. Experiments on three benchmarks, including theScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3Dconsistently outperforms current SOTA methods, even those specialized forindividual tasks. We hope UniSeg3D can serve as a solid unified baseline andinspire future work. The code will be available athttps://dk-liang.github.io/UniSeg3D/.