Abstract
We present a self-supervised approach to in-the-wild image relighting thatenables fully controllable, physically based illumination editing. We achievethis by combining the physical accuracy of traditional rendering with thephotorealistic appearance made possible by neural rendering. Our pipeline worksby inferring a colored mesh representation of a given scene using monocularestimates of geometry and intrinsic components. This representation allowsusers to define their desired illumination configuration in 3D. The scene underthe new lighting can then be rendered using a path-tracing engine. We send thisapproximate rendering of the scene through a feed-forward neural renderer topredict the final photorealistic relighting result. We develop a differentiablerendering process to reconstruct in-the-wild scene illumination, enablingself-supervised training of our neural renderer on raw image collections. Ourmethod represents a significant step in bringing the explicit physical controlover lights available in typical 3D computer graphics tools, such as Blender,to in-the-wild relighting.