Abstract
The light stage has been widely used in computer graphics for the past twodecades, primarily to enable the relighting of human faces. By capturing theappearance of the human subject under different light sources, one obtains thelight transport matrix of that subject, which enables image-based relighting innovel environments. However, due to the finite number of lights in the stage,the light transport matrix only represents a sparse sampling on the entiresphere. As a consequence, relighting the subject with a point light or adirectional source that does not coincide exactly with one of the lights in thestage requires interpolation and resampling the images corresponding to nearbylights, and this leads to ghosting shadows, aliased specularities, and otherartifacts. To ameliorate these artifacts and produce better results underarbitrary high-frequency lighting, this paper proposes a learning-basedsolution for the "super-resolution" of scans of human faces taken from a lightstage. Given an arbitrary "query" light direction, our method aggregates thecaptured images corresponding to neighboring lights in the stage, and uses aneural network to synthesize a rendering of the face that appears to beilluminated by a "virtual" light source at the query location. This neuralnetwork must circumvent the inherent aliasing and regularity of the light stagedata that was used for training, which we accomplish through the use ofregularized traditional interpolation methods within our network. Our learnedmodel is able to produce renderings for arbitrary light directions that exhibitrealistic shadows and specular highlights, and is able to generalize across awide variety of subjects.