Abstract
Neural Radiance Fields (NeRF) have recently gained a surge of interest withinthe computer vision community for its power to synthesize photorealistic novelviews of real-world scenes. One limitation of NeRF, however, is its requirementof accurate camera poses to learn the scene representations. In this paper, wepropose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF fromimperfect (or even unknown) camera poses -- the joint problem of learningneural 3D representations and registering camera frames. We establish atheoretical connection to classical image alignment and show thatcoarse-to-fine registration is also applicable to NeRF. Furthermore, we showthat na\"ively applying positional encoding in NeRF has a negative impact onregistration with a synthesis-based objective. Experiments on synthetic andreal-world data show that BARF can effectively optimize the neural scenerepresentations and resolve large camera pose misalignment at the same time.This enables view synthesis and localization of video sequences from unknowncamera poses, opening up new avenues for visual localization systems (e.g.SLAM) and potential applications for dense 3D mapping and reconstruction.