Abstract
We propose two novel ideas (adoption of deferred rendering and mesh-basedrepresentation) to improve the quality of 3D Gaussian splatting (3DGS) basedinverse rendering. We first report a problem incurred by hidden Gaussians,where Gaussians beneath the surface adversely affect the pixel color in thevolume rendering adopted by the existing methods. In order to resolve theproblem, we propose applying deferred rendering and report new problemsincurred in a naive application of deferred rendering to the existing3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-basedinverse rendering under deferred rendering, we propose a novel two-steptraining approach which (1) exploits mesh extraction and utilizes a hybridmesh-3DGS representation and (2) applies novel regularization methods to betterexploit the mesh. Our experiments show that, under relighting, the proposedmethod offers significantly better rendering quality than the existing3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-basedinverse rendering method, it gives better rendering quality while offeringreal-time rendering.