ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis

  • 2020-09-17 02:26:45
  • R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie
  • 33

Abstract

Manually authoring 3D shapes is difficult and time consuming; generativemodels of 3D shapes offer compelling alternatives. Procedural representationsare one such possibility: they offer high-quality and editable results but aredifficult to author and often produce outputs with limited diversity. On theother extreme are deep generative models: given enough data, they can learn togenerate any class of shape but their outputs have artifacts and therepresentation is not editable. In this paper, we take a step towards achievingthe best of both worlds for novel 3D shape synthesis. We propose ShapeAssembly,a domain-specific "assembly-language" for 3D shape structures. ShapeAssemblyprograms construct shapes by declaring cuboid part proxies and attaching themto one another, in a hierarchical and symmetrical fashion. Its functions areparameterized with free variables, so that one program structure is able tocapture a family of related shapes. We show how to extract ShapeAssemblyprograms from existing shape structures in the PartNet dataset. Then we train adeep generative model, a hierarchical sequence VAE, that learns to write novelShapeAssembly programs. The program captures the subset of variability that isinterpretable and editable. The deep model captures correlations across shapecollections that are hard to express procedurally. We evaluate our approach bycomparing shapes output by our generated programs to those from other recentshape structure synthesis models. We find that our generated shapes are moreplausible and physically-valid than those of other methods. Additionally, weassess the latent spaces of these models, and find that ours is betterstructured and produces smoother interpolations. As an application, we use ourgenerative model and differentiable program interpreter to infer and fit shapeprograms to unstructured geometry, such as point clouds.

 

Quick Read (beta)

loading the full paper ...