Generating the periodic structure of stable materials is a long-standingchallenge for the material design community. This task is difficult becausestable materials only exist in a low-dimensional subspace of all possibleperiodic arrangements of atoms: 1) the coordinates must lie in the local energyminimum defined by quantum mechanics, and 2) global stability also requires thestructure to follow the complex, yet specific bonding preferences betweendifferent atom types. Existing methods fail to incorporate these factors andoften lack proper invariances. We propose a Crystal Diffusion VariationalAutoencoder (CDVAE) that captures the physical inductive bias of materialstability. By learning from the data distribution of stable materials, thedecoder generates materials in a diffusion process that moves atomiccoordinates towards a lower energy state and updates atom types to satisfybonding preferences between neighbors. Our model also explicitly encodesinteractions across periodic boundaries and respects permutation, translation,rotation, and periodic invariances. We significantly outperform past methods inthree tasks: 1) reconstructing the input structure, 2) generating valid,diverse, and realistic materials, and 3) generating materials that optimize aspecific property. We also provide several standard datasets and evaluationmetrics for the broader machine learning community.