Synthesizing realistic images from human drawn sketches is a challengingproblem in computer graphics and vision. Existing approaches either need exactedge maps, or require a database to retrieve images from. In this work, wepropose a novel Generative Adversarial Network (GAN) approach that synthesizesrealistic looking images from 50 categories including motorcycles, horses andcouches. We demonstrate a data augmentation technique for sketches which isfully automatic, and we show that the augmented data is helpful to our task. Weintroduce a new building block suit for both the generator and discriminatorwhich improves the information flow and utilizes input images at multiplescales. Compared to state-of-the-art image translation methods, our approachgenerates more realistic images and achieves significantly higher InceptionScores.