Abstract
Semantic scene segmentation from a bird's-eye-view (BEV) perspective plays acrucial role in facilitating planning and decision-making for mobile robots.Although recent vision-only methods have demonstrated notable advancements inperformance, they often struggle under adverse illumination conditions such asrain or nighttime. While active sensors offer a solution to this challenge, theprohibitively high cost of LiDARs remains a limiting factor. Fusing camera datawith automotive radars poses a more inexpensive alternative but has receivedless attention in prior research. In this work, we aim to advance thispromising avenue by introducing BEVCar, a novel approach for joint BEV objectand map segmentation. The core novelty of our approach lies in first learning apoint-based encoding of raw radar data, which is then leveraged to efficientlyinitialize the lifting of image features into the BEV space. We performextensive experiments on the nuScenes dataset and demonstrate that BEVCaroutperforms the current state of the art. Moreover, we show that incorporatingradar information significantly enhances robustness in challengingenvironmental conditions and improves segmentation performance for distantobjects. To foster future research, we provide the weather split of thenuScenes dataset used in our experiments, along with our code and trainedmodels at http://bevcar.cs.uni-freiburg.de.