NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

  • 2021-11-26 18:59:47
  • Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T. Barron
  • 54

Abstract

Neural Radiance Fields (NeRF) is a technique for high quality novel viewsynthesis from a collection of posed input images. Like most view synthesismethods, NeRF uses tonemapped low dynamic range (LDR) as input; these imageshave been processed by a lossy camera pipeline that smooths detail, clipshighlights, and distorts the simple noise distribution of raw sensor data. Wemodify NeRF to instead train directly on linear raw images, preserving thescene's full dynamic range. By rendering raw output images from the resultingNeRF, we can perform novel high dynamic range (HDR) view synthesis tasks. Inaddition to changing the camera viewpoint, we can manipulate focus, exposure,and tonemapping after the fact. Although a single raw image appearssignificantly more noisy than a postprocessed one, we show that NeRF is highlyrobust to the zero-mean distribution of raw noise. When optimized over manynoisy raw inputs (25-200), NeRF produces a scene representation so accuratethat its rendered novel views outperform dedicated single and multi-image deepraw denoisers run on the same wide baseline input images. As a result, ourmethod, which we call RawNeRF, can reconstruct scenes from extremely noisyimages captured in near-darkness.

 

Quick Read (beta)

loading the full paper ...