Abstract
Casting semantic segmentation of outdoor LiDAR point clouds as a 2D problem,e.g., via range projection, is an effective and popular approach. Theseprojection-based methods usually benefit from fast computations and, whencombined with techniques which use other point cloud representations, achievestate-of-the-art results. Today, projection-based methods leverage 2D CNNs butrecent advances in computer vision show that vision transformers (ViTs) haveachieved state-of-the-art results in many image-based benchmarks. In this work,we question if projection-based methods for 3D semantic segmentation canbenefit from these latest improvements on ViTs. We answer positively but onlyafter combining them with three key ingredients: (a) ViTs are notoriously hardto train and require a lot of training data to learn powerful representations.By preserving the same backbone architecture as for RGB images, we can exploitthe knowledge from long training on large image collections that are muchcheaper to acquire and annotate than point clouds. We reach our best resultswith pre-trained ViTs on large image datasets. (b) We compensate ViTs' lack ofinductive bias by substituting a tailored convolutional stem for the classicallinear embedding layer. (c) We refine pixel-wise predictions with aconvolutional decoder and a skip connection from the convolutional stem tocombine low-level but fine-grained features of the the convolutional stem withthe high-level but coarse predictions of the ViT encoder. With theseingredients, we show that our method, called RangeViT, outperforms existingprojection-based methods on nuScenes and SemanticKITTI. We provide theimplementation code at https://github.com/valeoai/rangevit.