Abstract
As previous representations for reinforcement learning cannot effectivelyincorporate a human-intuitive understanding of the 3D environment, they usuallysuffer from sub-optimal performances. In this paper, we present Semantic-awareNeural Radiance Fields for Reinforcement Learning (SNeRL), which jointlyoptimizes semantic-aware neural radiance fields (NeRF) with a convolutionalencoder to learn 3D-aware neural implicit representation from multi-viewimages. We introduce 3D semantic and distilled feature fields in parallel tothe RGB radiance fields in NeRF to learn semantic and object-centricrepresentation for reinforcement learning. SNeRL outperforms not only previouspixel-based representations but also recent 3D-aware representations both inmodel-free and model-based reinforcement learning.