Text-To-4D Dynamic Scene Generation

  • 2023-01-26 18:14:32
  • Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, Yaniv Taigman
  • 254


We present MAV3D (Make-A-Video3D), a method for generating three-dimensionaldynamic scenes from text descriptions. Our approach uses a 4D dynamic NeuralRadiance Field (NeRF), which is optimized for scene appearance, density, andmotion consistency by querying a Text-to-Video (T2V) diffusion-based model. Thedynamic video output generated from the provided text can be viewed from anycamera location and angle, and can be composited into any 3D environment. MAV3Ddoes not require any 3D or 4D data and the T2V model is trained only onText-Image pairs and unlabeled videos. We demonstrate the effectiveness of ourapproach using comprehensive quantitative and qualitative experiments and showan improvement over previously established internal baselines. To the best ofour knowledge, our method is the first to generate 3D dynamic scenes given atext description.


Quick Read (beta)

loading the full paper ...