Abstract
We investigate the problem of Language-Based Image Editing (LBIE). Given asource image and a natural language description, we want to generate a targetimage by editing the source image based on the description. We propose ageneric modeling framework for two sub-tasks of LBIE: language-based imagesegmentation and image colorization. The framework uses recurrent attentivemodels to fuse image and language features. Instead of using a fixed step size,we introduce for each region of the image a termination gate to dynamicallydetermine after each inference step whether to continue extrapolatingadditional information from the textual description. The effectiveness of theframework is validated on three datasets. First, we introduce a syntheticdataset, called CoSaL, to evaluate the end-to-end performance of our LBIEsystem. Second, we show that the framework leads to state-of-the-artperformance on image segmentation on the ReferIt dataset. Third, we present thefirst language-based colorization result on the Oxford-102 Flowers dataset.