Diffusion-based Generation, Optimization, and Planning in 3D Scenes

  • 2023-01-15 03:43:45
  • Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, Song-Chun Zhu
  • 40


We introduce SceneDiffuser, a conditional generative model for 3D sceneunderstanding. SceneDiffuser provides a unified model for solvingscene-conditioned generation, optimization, and planning. In contrast to priorworks, SceneDiffuser is intrinsically scene-aware, physics-based, andgoal-oriented. With an iterative sampling strategy, SceneDiffuser jointlyformulates the scene-aware generation, physics-based optimization, andgoal-oriented planning via a diffusion-based denoising process in a fullydifferentiable fashion. Such a design alleviates the discrepancies amongdifferent modules and the posterior collapse of previous scene-conditionedgenerative models. We evaluate SceneDiffuser with various 3D sceneunderstanding tasks, including human pose and motion generation, dexterousgrasp generation, path planning for 3D navigation, and motion planning forrobot arms. The results show significant improvements compared with previousmodels, demonstrating the tremendous potential of SceneDiffuser for the broadcommunity of 3D scene understanding.


Quick Read (beta)

loading the full paper ...