GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

  • 2021-04-15 17:59:38
  • Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu
  • 130

Abstract

We present GANcraft, an unsupervised neural rendering framework forgenerating photorealistic images of large 3D block worlds such as those createdin Minecraft. Our method takes a semantic block world as input, where eachblock is assigned a semantic label such as dirt, grass, or water. We representthe world as a continuous volumetric function and train our model to renderview-consistent photorealistic images for a user-controlled camera. In theabsence of paired ground truth real images for the block world, we devise atraining technique based on pseudo-ground truth and adversarial training. Thisstands in contrast to prior work on neural rendering for view synthesis, whichrequires ground truth images to estimate scene geometry and view-dependentappearance. In addition to camera trajectory, GANcraft allows user control overboth scene semantics and output style. Experimental results with comparison tostrong baselines show the effectiveness of GANcraft on this novel task ofphotorealistic 3D block world synthesis. The project website is available athttps://nvlabs.github.io/GANcraft/ .

 

Quick Read (beta)

loading the full paper ...