Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches

  • 2020-01-09 08:57:50
  • Shuai Yang, Zhangyang Wang, Jiaying Liu, Zongming Guo
  • 36

Abstract

Sketch-based image editing aims to synthesize and modify photos based on thestructural information provided by the human-drawn sketches. Since sketches aredifficult to collect, previous methods mainly use edge maps instead of sketchesto train models (referred to as edge-based models). However, sketches displaygreat structural discrepancy with edge maps, thus failing edge-based models.Moreover, sketches often demonstrate huge variety among different users,demanding even higher generalizability and robustness for the editing model towork. In this paper, we propose Deep Plastic Surgery, a novel, robust andcontrollable image editing framework that allows users to interactively editimages using hand-drawn sketch inputs. We present a sketch refinement strategy,as inspired by the coarse-to-fine drawing process of the artists, which we showcan help our model well adapt to casual and varied sketches without the needfor real sketch training data. Our model further provides a refinement levelcontrol parameter that enables users to flexibly define how "reliable" theinput sketch should be considered for the final output, balancing betweensketch faithfulness and output verisimilitude (as the two goals mightcontradict if the input sketch is drawn poorly). To achieve the multi-levelrefinement, we introduce a style-based module for level conditioning, whichallows adaptive feature representations for different levels in a singenetwork. Extensive experimental results demonstrate the superiority of ourapproach in improving the visual quality and user controllablity of imageediting over the state-of-the-art methods.

 

Quick Read (beta)

loading the full paper ...