Abstract
We propose the Large View Synthesis Model (LVSM), a novel transformer-basedapproach for scalable and generalizable novel view synthesis from sparse-viewinputs. We introduce two architectures: (1) an encoder-decoder LVSM, whichencodes input image tokens into a fixed number of 1D latent tokens, functioningas a fully learned scene representation, and decodes novel-view images fromthem; and (2) a decoder-only LVSM, which directly maps input images tonovel-view outputs, completely eliminating intermediate scene representations.Both models bypass the 3D inductive biases used in previous methods -- from 3Drepresentations (e.g., NeRF, 3DGS) to network designs (e.g., epipolarprojections, plane sweeps) -- addressing novel view synthesis with a fullydata-driven approach. While the encoder-decoder model offers faster inferencedue to its independent latent representation, the decoder-only LVSM achievessuperior quality, scalability, and zero-shot generalization, outperformingprevious state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensiveevaluations across multiple datasets demonstrate that both LVSM variantsachieve state-of-the-art novel view synthesis quality. Notably, our modelssurpass all previous methods even with reduced computational resources (1-2GPUs). Please see our website for more details:https://haian-jin.github.io/projects/LVSM/ .