Abstract
Recent techniques for real-time view synthesis have rapidly advanced infidelity and speed, and modern methods are capable of renderingnear-photorealistic scenes at interactive frame rates. At the same time, atension has arisen between explicit scene representations amenable torasterization and neural fields built on ray marching, with state-of-the-artinstances of the latter surpassing the former in quality while beingprohibitively expensive for real-time applications. In this work, we introduceSMERF, a view synthesis approach that achieves state-of-the-art accuracy amongreal-time methods on large scenes with footprints up to 300 m$^2$ at avolumetric resolution of 3.5 mm$^3$. Our method is built upon two primarycontributions: a hierarchical model partitioning scheme, which increases modelcapacity while constraining compute and memory consumption, and a distillationtraining strategy that simultaneously yields high fidelity and internalconsistency. Our approach enables full six degrees of freedom (6DOF) navigationwithin a web browser and renders in real-time on commodity smartphones andlaptops. Extensive experiments show that our method exceeds the currentstate-of-the-art in real-time novel view synthesis by 0.78 dB on standardbenchmarks and 1.78 dB on large scenes, renders frames three orders ofmagnitude faster than state-of-the-art radiance field models, and achievesreal-time performance across a wide variety of commodity devices, includingsmartphones. We encourage the reader to explore these models in person at ourproject website: https://smerf-3d.github.io.