Abstract
Existing approaches to system identification (estimating the physicalparameters of an object) from videos assume known object geometries. Thisprecludes their applicability in a vast majority of scenes where objectgeometries are complex or unknown. In this work, we aim to identify parameterscharacterizing a physical system from a set of multi-view videos without anyassumption on object geometry or topology. To this end, we propose "PhysicsAugmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both theunknown geometry and physical parameters of highly dynamic objects frommulti-view videos. We design PAC-NeRF to only ever produce physically plausiblestates by enforcing the neural radiance field to follow the conservation lawsof continuum mechanics. For this, we design a hybrid Eulerian-Lagrangianrepresentation of the neural radiance field, i.e., we use the Eulerian gridrepresentation for NeRF density and color fields, while advecting the neuralradiance fields via Lagrangian particles. This hybrid Eulerian-Lagrangianrepresentation seamlessly blends efficient neural rendering with the materialpoint method (MPM) for robust differentiable physics simulation. We validatethe effectiveness of our proposed framework on geometry and physical parameterestimation over a vast range of materials, including elastic bodies,plasticine, sand, Newtonian and non-Newtonian fluids, and demonstratesignificant performance gain on most tasks.