Abstract
Neural rendering has made significant strides in 3D reconstruction and novelview synthesis. With the integration with physics, it opens up newapplications. The inverse problem of estimating physics from visual data,however, still remains challenging, limiting its effectiveness for applicationslike physically accurate digital twin creation in robotics and XR. Existingmethods that incorporate physics into neural rendering frameworks typicallyrequire dense multi-view videos as input, making them impractical for scalable,real-world use. When presented with sparse multi-view videos, the sequentialoptimization strategy used by existing approaches introduces significant erroraccumulation, e.g., poor initial 3D reconstruction leads to bad materialparameter estimation in subsequent stages. Instead of sequential optimization,directly optimizing all parameters at the same time also fails due to thehighly non-convex and often non-differentiable nature of the problem. Wepropose ProJo4D, a progressive joint optimization framework that graduallyincreases the set of jointly optimized parameters guided by their sensitivity,leading to fully joint optimization over geometry, appearance, physical state,and material property. Evaluations on PAC-NeRF and Spring-Gaus datasets showthat ProJo4D outperforms prior work in 4D future state prediction, novel viewrendering of future state, and material parameter estimation, demonstrating itseffectiveness in physically grounded 4D scene understanding. For demos, pleasevisit the project webpage: https://daniel03c1.github.io/ProJo4D/