Abstract
Neural fields excel in computer vision and robotics due to their ability tounderstand the 3D visual world such as inferring semantics, geometry, anddynamics. Given the capabilities of neural fields in densely representing a 3Dscene from 2D images, we ask the question: Can we scale their self-supervisedpretraining, specifically using masked autoencoders, to generate effective 3Drepresentations from posed RGB images. Owing to the astounding success ofextending transformers to novel data modalities, we employ standard 3D VisionTransformers to suit the unique formulation of NeRFs. We leverage NeRF'svolumetric grid as a dense input to the transformer, contrasting it with other3D representations such as pointclouds where the information density can beuneven, and the representation is irregular. Due to the difficulty of applyingmasked autoencoders to an implicit representation, such as NeRF, we opt forextracting an explicit representation that canonicalizes scenes across domainsby employing the camera trajectory for sampling. Our goal is made possible bymasking random patches from NeRF's radiance and density grid and employing astandard 3D Swin Transformer to reconstruct the masked patches. In doing so,the model can learn the semantic and spatial structure of complete scenes. Wepretrain this representation at scale on our proposed curated posed-RGB data,totaling over 1.6 million images. Once pretrained, the encoder is used foreffective 3D transfer learning. Our novel self-supervised pretraining forNeRFs, NeRF-MAE, scales remarkably well and improves performance on variouschallenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRFscene understanding baselines on Front3D and ScanNet datasets with an absoluteperformance improvement of over 20% AP50 and 8% AP25 for 3D object detection.