Abstract
Modeling the dynamics of deformable objects is challenging due to theirdiverse physical properties and the difficulty of estimating states fromlimited visual information. We address these challenges with a neural dynamicsframework that combines object particles and spatial grids in a hybridrepresentation. Our particle-grid model captures global shape and motioninformation while predicting dense particle movements, enabling the modeling ofobjects with varied shapes and materials. Particles represent object shapes,while the spatial grid discretizes the 3D space to ensure spatial continuityand enhance learning efficiency. Coupled with Gaussian Splattings for visualrendering, our framework achieves a fully learning-based digital twin ofdeformable objects and generates 3D action-conditioned videos. Throughexperiments, we demonstrate that our model learns the dynamics of diverseobjects -- such as ropes, cloths, stuffed animals, and paper bags -- fromsparse-view RGB-D recordings of robot-object interactions, while alsogeneralizing at the category level to unseen instances. Our approachoutperforms state-of-the-art learning-based and physics-based simulators,particularly in scenarios with limited camera views. Furthermore, we showcasethe utility of our learned models in model-based planning, enablinggoal-conditioned object manipulation across a range of tasks. The project pageis available at https://kywind.github.io/pgnd .