Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

  • 2022-11-14 18:25:24
  • Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, Daniel Cohen-Or
  • 54


Text-guided image generation has progressed rapidly in recent years,inspiring major breakthroughs in text-guided shape generation. Recently, it hasbeen shown that using score distillation, one can successfully text-guide aNeRF model to generate a 3D object. We adapt the score distillation to thepublicly available, and computationally efficient, Latent Diffusion Models,which apply the entire diffusion process in a compact latent space of apretrained autoencoder. As NeRFs operate in image space, a naive solution forguiding them with latent score distillation would require encoding to thelatent space at each guidance step. Instead, we propose to bring the NeRF tothe latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, weshow that while Text-to-3D models can generate impressive results, they areinherently unconstrained and may lack the ability to guide or enforce aspecific 3D structure. To assist and direct the 3D generation, we propose toguide our Latent-NeRF using a Sketch-Shape: an abstract geometry that definesthe coarse structure of the desired object. Then, we present means to integratesuch a constraint directly into a Latent-NeRF. This unique combination of textand shape guidance allows for increased control over the generation process. Wealso show that latent score distillation can be successfully applied directlyon 3D meshes. This allows for generating high-quality textures on a givengeometry. Our experiments validate the power of our different forms of guidanceand the efficiency of using latent rendering. Implementation is available at


Quick Read (beta)

loading the full paper ...