Abstract
Sketch-based image manipulation is an interactive image editing task tomodify an image based on input sketches from users. Existing methods typicallyformulate this task as a conditional inpainting problem, which requires usersto draw an extra mask indicating the region to modify in addition to sketches.The masked regions are regarded as holes and filled by an inpainting modelconditioned on the sketch. With this formulation, paired training data can beeasily obtained by randomly creating masks and extracting edges or contours.Although this setup simplifies data preparation and model design, itcomplicates user interaction and discards useful information in masked regions.To this end, we investigate a new paradigm of sketch-based image manipulation:mask-free local image manipulation, which only requires sketch inputs fromusers and utilizes the entire original image. Given an image and sketch, ourmodel automatically predicts the target modification region and encodes it intoa structure agnostic style vector. A generator then synthesizes the new imagecontent based on the style vector and sketch. The manipulated image is finallyproduced by blending the generator output into the modification region of theoriginal image. Our model can be trained in a self-supervised fashion bylearning the reconstruction of an image region from the style vector andsketch. The proposed method offers simpler and more intuitive user workflowsfor sketch-based image manipulation and provides better results than previousapproaches. More results, code and interactive demo will be available at\url{https://zengxianyu.github.io/sketchedit}.