Abstract
Single-image relighting is a challenging task that involves reasoning aboutthe complex interplay between geometry, materials, and lighting. Many priormethods either support only specific categories of images, such as portraits,or require special capture conditions, like using a flashlight. Alternatively,some methods explicitly decompose a scene into intrinsic components, such asnormals and BRDFs, which can be inaccurate or under-expressive. In this work,we propose a novel end-to-end 2D relighting diffusion model, called NeuralGaffer, that takes a single image of any object and can synthesize an accurate,high-quality relit image under any novel environmental lighting condition,simply by conditioning an image generator on a target environment map, withoutan explicit scene decomposition. Our method builds on a pre-trained diffusionmodel, and fine-tunes it on a synthetic relighting dataset, revealing andharnessing the inherent understanding of lighting present in the diffusionmodel. We evaluate our model on both synthetic and in-the-wild Internet imageryand demonstrate its advantages in terms of generalization and accuracy.Moreover, by combining with other generative methods, our model enables manydownstream 2D tasks, such as text-based relighting and object insertion. Ourmodel can also operate as a strong relighting prior for 3D tasks, such asrelighting a radiance field.