Abstract
3D scene understanding plays a vital role in vision-based autonomous driving.While most existing methods focus on 3D object detection, they have difficultydescribing real-world objects of arbitrary shapes and infinite classes. Towardsa more comprehensive perception of a 3D scene, in this paper, we propose aSurroundOcc method to predict the 3D occupancy with multi-camera images. Wefirst extract multi-scale features for each image and adopt spatial 2D-3Dattention to lift them to the 3D volume space. Then we apply 3D convolutions toprogressively upsample the volume features and impose supervision on multiplelevels. To obtain dense occupancy prediction, we design a pipeline to generatedense occupancy ground truth without expansive occupancy annotations.Specifically, we fuse multi-frame LiDAR scans of dynamic objects and staticscenes separately. Then we adopt Poisson Reconstruction to fill the holes andvoxelize the mesh to get dense occupancy labels. Extensive experiments onnuScenes and SemanticKITTI datasets demonstrate the superiority of our method.Code and dataset are available at https://github.com/weiyithu/SurroundOcc