Abstract
Given a portrait image of a person and an environment map of the targetlighting, portrait relighting aims to re-illuminate the person in the image asif the person appeared in an environment with the target lighting. To achievehigh-quality results, recent methods rely on deep learning. An effectiveapproach is to supervise the training of deep neural networks with ahigh-fidelity dataset of desired input-output pairs, captured with a lightstage. However, acquiring such data requires an expensive special capture rigand time-consuming efforts, limiting access to only a few resourcefullaboratories. To address the limitation, we propose a new approach that canperform on par with the state-of-the-art (SOTA) relighting methods withoutrequiring a light stage. Our approach is based on the realization that asuccessful relighting of a portrait image depends on two conditions. First, themethod needs to mimic the behaviors of physically-based relighting. Second, theoutput has to be photorealistic. To meet the first condition, we propose totrain the relighting network with training data generated by a virtual lightstage that performs physically-based rendering on various 3D synthetic humansunder different environment maps. To meet the second condition, we develop anovel synthetic-to-real approach to bring photorealism to the relightingnetwork output. In addition to achieving SOTA results, our approach offersseveral advantages over the prior methods, including controllable glares onglasses and more temporally-consistent results for relighting videos.