Abstract
Despite significant recent progress on generative models, controlledgeneration of images depicting multiple and complex object layouts is still adifficult problem. Among the core challenges are the diversity of appearance agiven object may possess and, as a result, exponential set of images consistentwith a specified layout. To address these challenges, we propose a novelapproach for layout-based image generation; we call it Layout2Im. Given thecoarse spatial layout (bounding boxes + object categories), our model cangenerate a set of realistic images which have the correct objects in thedesired locations. The representation of each object is disentangled into aspecified/certain part (category) and an unspecified/uncertain part(appearance). The category is encoded using a word embedding and the appearanceis distilled into a low-dimensional vector sampled from a normal distribution.Individual object representations are composed together using convolutionalLSTM, to obtain an encoding of the complete layout, and then decoded to animage. Several loss terms are introduced to encourage accurate and diversegeneration. The proposed Layout2Im model significantly outperforms the previousstate of the art, boosting the best reported inception score by 24.66% and28.57% on the very challenging COCO-Stuff and Visual Genome datasets,respectively. Extensive experiments also demonstrate our method's ability togenerate complex and diverse images with multiple objects.