Abstract
A key challenge in fine-grained 3D-based interactive editing is the absenceof an efficient representation that balances diverse modifications withhigh-quality view synthesis under a given memory constraint. While 3D meshesprovide robustness for various modifications, they often yield lower-qualityview synthesis compared to 3D Gaussian Splatting, which, in turn, suffers frominstability during extensive editing. A straightforward combination of thesetwo representations results in suboptimal performance and fails to meet memoryconstraints. In this paper, we introduce SplatMesh, a novel fine-grainedinteractive 3D segmentation and editing algorithm that integrates 3D GaussianSplat with a precomputed mesh and could adjust the memory request based on therequirement. Specifically, given a mesh, \method simplifies it whileconsidering both color and shape, ensuring it meets memory constraints. Then,SplatMesh aligns Gaussian splats with the simplified mesh by treating eachtriangle as a new reference point. By segmenting and editing the simplifiedmesh, we can effectively edit the Gaussian splats as well, which will lead toextensive experiments on real and synthetic datasets, coupled with illustrativevisual examples, highlighting the superiority of our approach in terms ofrepresentation quality and editing performance. Code of our paper can be foundhere: https://github.com/kaichen-z/SplatMesh.