Abstract
The goal of this paper is to augment a pre-trained text-to-image diffusionmodel with the ability of open-vocabulary objects grounding, i.e.,simultaneously generating images and segmentation masks for the correspondingvisual entities described in the text prompt. We make the followingcontributions: (i) we insert a grounding module into the existing diffusionmodel, that can be trained to align the visual and textual embedding space ofthe diffusion model with only a small number of object categories; (ii) wepropose an automatic pipeline for constructing a dataset, that consists of{image, segmentation mask, text prompt} triplets, to train the proposedgrounding module; (iii) we evaluate the performance of open-vocabularygrounding on images generated from the text-to-image diffusion model and showthat the module can well segment the objects of categories beyond seen ones attraining time; (iv) we adopt the guided diffusion model to build a syntheticsemantic segmentation dataset, and show that training a standard segmentationmodel on such dataset demonstrates competitive performance on zero-shotsegmentation(ZS3) benchmark, which opens up new opportunities for adopting thepowerful diffusion model for discriminative tasks.