Abstract
Latent 3D reconstruction has shown great promise in empowering 3D semanticunderstanding and 3D generation by distilling 2D features into the 3D space.However, existing approaches struggle with the domain gap between 2D featurespace and 3D representations, resulting in degraded rendering performance. Toaddress this challenge, we propose a novel framework that integrates 3Dawareness into the 2D latent space. The framework consists of three stages: (1)a correspondence-aware autoencoding method that enhances the 3D consistency of2D latent representations, (2) a latent radiance field (LRF) that lifts these3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field(VAE-RF) alignment strategy that improves image decoding from the rendered 2Drepresentations. Extensive experiments demonstrate that our method outperformsthe state-of-the-art latent 3D reconstruction approaches in terms of synthesisperformance and cross-dataset generalizability across diverse indoor andoutdoor scenes. To our knowledge, this is the first work showing the radiancefield representations constructed from 2D latent representations can yieldphotorealistic 3D reconstruction performance.