SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields

  • 2022-11-22 13:14:50
  • Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, Alex Levinshtein
  • 58


Neural Radiance Fields (NeRFs) have emerged as a popular approach for novelview synthesis. While NeRFs are quickly being adapted for a wider set ofapplications, intuitively editing NeRF scenes is still an open challenge. Oneimportant editing task is the removal of unwanted objects from a 3D scene, suchthat the replaced region is visually plausible and consistent with its context.We refer to this task as 3D inpainting. In 3D, solutions must be bothconsistent across multiple views and geometrically valid. In this paper, wepropose a novel 3D inpainting method that addresses these challenges. Given asmall set of posed images and sparse annotations in a single input image, ourframework first rapidly obtains a 3D segmentation mask for a target object.Using the mask, a perceptual optimizationbased approach is then introduced thatleverages learned 2D image inpainters, distilling their information into 3Dspace, while ensuring view consistency. We also address the lack of a diversebenchmark for evaluating 3D scene inpainting methods by introducing a datasetcomprised of challenging real-world scenes. In particular, our dataset containsviews of the same scene with and without a target object, enabling moreprincipled benchmarking of the 3D inpainting task. We first demonstrate thesuperiority of our approach on multiview segmentation, comparing to NeRFbasedmethods and 2D segmentation approaches. We then evaluate on the task of 3Dinpainting, establishing state-ofthe-art performance against other NeRFmanipulation algorithms, as well as a strong 2D image inpainter baseline


Quick Read (beta)

loading the full paper ...