Abstract
Vision foundation models (VFMs) trained on large-scale image datasets providehigh-quality features that have significantly advanced 2D visual recognition.However, their potential in 3D vision remains largely untapped, despite thecommon availability of 2D images alongside 3D point cloud datasets. Whilesignificant research has been dedicated to 2D-3D fusion, recentstate-of-the-art 3D methods predominantly focus on 3D data, leaving theintegration of VFMs into 3D models underexplored. In this work, we challengethis trend by introducing DITR, a simple yet effective approach that extracts2D foundation model features, projects them to 3D, and finally injects theminto a 3D point cloud segmentation model. DITR achieves state-of-the-artresults on both indoor and outdoor 3D semantic segmentation benchmarks. Toenable the use of VFMs even when images are unavailable during inference, wefurther propose to distill 2D foundation models into a 3D backbone as apretraining task. By initializing the 3D backbone with knowledge distilled from2D VFMs, we create a strong basis for downstream 3D segmentation tasks,ultimately boosting performance across various datasets.