3D Ken Burns Effect from a Single Image

  • 2019-09-12 06:55:07
  • Simon Niklaus, Long Mai, Jimei Yang, Feng Liu
  • 227

Abstract

The Ken Burns effect allows animating still images with a virtual camera scanand zoom. Adding parallax, which results in the 3D Ken Burns effect, enablessignificantly more compelling results. Creating such effects manually istime-consuming and demands sophisticated editing skills. Existing automaticmethods, however, require multiple input images from varying viewpoints. Inthis paper, we introduce a framework that synthesizes the 3D Ken Burns effectfrom a single image, supporting both a fully automatic mode and an interactivemode with the user controlling the camera. Our framework first leverages adepth prediction pipeline, which estimates scene depth that is suitable forview synthesis tasks. To address the limitations of existing depth estimationmethods such as geometric distortions, semantic distortions, and inaccuratedepth boundaries, we develop a semantic-aware neural network for depthprediction, couple its estimate with a segmentation-based depth adjustmentprocess, and employ a refinement neural network that facilitates accurate depthpredictions at object boundaries. According to this depth estimate, ourframework then maps the input image to a point cloud and synthesizes theresulting video frames by rendering the point cloud from the correspondingcamera positions. To address disocclusions while maintaining geometrically andtemporally coherent synthesis results, we utilize context-aware color- anddepth-inpainting to fill in the missing information in the extreme views of thecamera path, thus extending the scene geometry of the point cloud. Experimentswith a wide variety of image content show that our method enables realisticsynthesis results. Our study demonstrates that our system allows users toachieve better results while requiring little effort compared to existingsolutions for the 3D Ken Burns effect creation.

 

Quick Read (beta)

loading the full paper ...