Abstract
We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scenereconstruction, trained entirely on real-world monocular posed videos. Using 4DGaussian as an inductive bias, 4DGT unifies static and dynamic components,enabling the modeling of complex, time-varying environments with varying objectlifespans. We proposed a novel density control strategy in training, whichenables our 4DGT to handle longer space-time input and remain efficientrendering at runtime. Our model processes 64 consecutive posed frames in arolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlikeoptimization-based methods, 4DGT performs purely feed-forward inference,reducing reconstruction time from hours to seconds and scaling effectively tolong video sequences. Trained only on large-scale monocular posed videodatasets, 4DGT can outperform prior Gaussian-based networks significantly inreal-world videos and achieve on-par accuracy with optimization-based methodson cross-domain videos. Project page: https://4dgt.github.io