We present a deep learning solution for estimating the incident illuminationat any 3D location within a scene from an input narrow-baseline stereo imagepair. Previous approaches for predicting global illumination from images eitherpredict just a single illumination for the entire scene, or separately estimatethe illumination at each 3D location without enforcing that the predictions areconsistent with the same 3D scene. Instead, we propose a deep learning modelthat estimates a 3D volumetric RGBA model of a scene, including content outsidethe observed field of view, and then uses standard volume rendering to estimatethe incident illumination at any 3D location within that volume. Our model istrained without any ground truth 3D data and only requires a held-outperspective view near the input stereo pair and a spherical panorama takenwithin each scene as supervision, as opposed to prior methods forspatially-varying lighting estimation, which require ground truth scenegeometry for training. We demonstrate that our method can predict consistentspatially-varying lighting that is convincing enough to plausibly relight andinsert highly specular virtual objects into real images.