Abstract
Existing reconstruction-based novel view synthesis methods for driving scenesfocus on synthesizing camera views along the recorded trajectory of the egovehicle. Their image rendering performance will severely degrade on viewpointsfalling out of the recorded trajectory, where camera rays are untrained. Wepropose FreeVS, a novel fully generative approach that can synthesize cameraviews on free new trajectories in real driving scenes. To control thegeneration results to be 3D consistent with the real scenes and accurate inviewpoint pose, we propose the pseudo-image representation of view priors tocontrol the generation process. Viewpoint transformation simulation is appliedon pseudo-images to simulate camera movement in each direction. Once trained,FreeVS can be applied to any validation sequences without reconstructionprocess and synthesis views on novel trajectories. Moreover, we propose two newchallenging benchmarks tailored to driving scenes, which are novel camerasynthesis and novel trajectory synthesis, emphasizing the freedom ofviewpoints. Given that no ground truth images are available on noveltrajectories, we also propose to evaluate the consistency of images synthesizedon novel trajectories with 3D perception models. Experiments on the Waymo OpenDataset show that FreeVS has a strong image synthesis performance on both therecorded trajectories and novel trajectories. Project Page:https://freevs24.github.io/