Abstract
Understanding 3D scenes from a single image is fundamental to a wide varietyof tasks, such as for robotics, motion planning, or augmented reality. Existingworks in 3D perception from a single RGB image tend to focus on geometricreconstruction only, or geometric reconstruction with semantic segmentation orinstance segmentation. Inspired by 2D panoptic segmentation, we propose tounify the tasks of geometric reconstruction, 3D semantic segmentation, and 3Dinstance segmentation into the task of panoptic 3D scene reconstruction - froma single RGB image, predicting the complete geometric reconstruction of thescene in the camera frustum of the image, along with semantic and instancesegmentations. We thus propose a new approach for holistic 3D sceneunderstanding from a single RGB image which learns to lift and propagate 2Dfeatures from an input image to a 3D volumetric scene representation. Wedemonstrate that this holistic view of joint scene reconstruction, semantic,and instance segmentation is beneficial over treating the tasks independently,thus outperforming alternative approaches.