Abstract
Sketch-based image synthesis aims to generate a photo image given a sketch.It is a challenging task; because sketches are drawn by non-professionals andonly consist of strokes, they usually exhibit shape deformation and lack visualcues, i.e., colors and textures. Thus translation from sketch to photo involvestwo aspects: shape and color (texture). Existing methods cannot handle thistask well, as they mostly focus on solving one translation. In this work, weshow that the key to this task lies in decomposing the translation into twosub-tasks, shape translation and colorization. Correspondingly, we propose amodel consisting of two sub-networks, with each one tackling one sub-task. Wealso find that, when translating shapes, specific drawing styles affect thegenerated results significantly and may even lead to failure. To make our modelmore robust to drawing style variations, we design a data augmentation strategyand re-purpose an attention module, aiming to make our model pay less attentionto distracted regions of a sketch. Besides, a conditional module is adapted forcolor translation to improve diversity and increase users' control over thegenerated results. Both quantitative and qualitative comparisons are presentedto show the superiority of our approach. In addition, as a side benefit, ourmodel can synthesize high-quality sketches from photos inversely. We alsodemonstrate how these generated photos and sketches can benefit otherapplications, such as sketch-based image retrieval.