SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning

  • 2023-05-31 17:21:18
  • Dongseok Shim, Seungjae Lee, H. Jin Kim
  • 0


As previous representations for reinforcement learning cannot effectivelyincorporate a human-intuitive understanding of the 3D environment, they usuallysuffer from sub-optimal performances. In this paper, we present Semantic-awareNeural Radiance Fields for Reinforcement Learning (SNeRL), which jointlyoptimizes semantic-aware neural radiance fields (NeRF) with a convolutionalencoder to learn 3D-aware neural implicit representation from multi-viewimages. We introduce 3D semantic and distilled feature fields in parallel tothe RGB radiance fields in NeRF to learn semantic and object-centricrepresentation for reinforcement learning. SNeRL outperforms not only previouspixel-based representations but also recent 3D-aware representations both inmodel-free and model-based reinforcement learning.


Quick Read (beta)

loading the full paper ...