Recognizing Scenes from Novel Viewpoints

  • 2021-12-02 18:59:40
  • Shengyi Qian, Alexander Kirillov, Nikhila Ravi, Devendra Singh Chaplot, Justin Johnson, David F. Fouhey, Georgia Gkioxari
  • 15

Abstract

Humans can perceive scenes in 3D from a handful of 2D views. For AI agents,the ability to recognize a scene from any viewpoint given only a few imagesenables them to efficiently interact with the scene and its objects. In thiswork, we attempt to endow machines with this ability. We propose a model whichtakes as input a few RGB images of a new scene and recognizes the scene fromnovel viewpoints by segmenting it into semantic categories. All this withoutaccess to the RGB images from those views. We pair 2D scene recognition with animplicit 3D representation and learn from multi-view 2D annotations of hundredsof scenes without any 3D supervision beyond camera poses. We experiment onchallenging datasets and demonstrate our model's ability to jointly capturesemantics and geometry of novel scenes with diverse layouts, object types andshapes.

 

Quick Read (beta)

loading the full paper ...