Abstract
We present Stable View Synthesis (SVS). Given a set of source imagesdepicting a scene from freely distributed viewpoints, SVS synthesizes new viewsof the scene. The method operates on a geometric scaffold computed viastructure-from-motion and multi-view stereo. Each point on this 3D scaffold isassociated with view rays and corresponding feature vectors that encode theappearance of this point in the input images. The core of SVS is view-dependenton-surface feature aggregation, in which directional feature vectors at each 3Dpoint are processed to produce a new feature vector for a ray that maps thispoint into the new target view. The target view is then rendered by aconvolutional network from a tensor of features synthesized in this way for allpixels. The method is composed of differentiable modules and is trainedend-to-end. It supports spatially-varying view-dependent importance weightingand feature transformation of source images at each point; spatial and temporalstability due to the smooth dependence of on-surface feature aggregation on thetarget view; and synthesis of view-dependent effects such as specularreflection. Experimental results demonstrate that SVS outperformsstate-of-the-art view synthesis methods both quantitatively and qualitativelyon three diverse real-world datasets, achieving unprecedented levels of realismin free-viewpoint video of challenging large-scale scenes.