### Abstract

Images with shared characteristics naturally form sets. For example, in aface verification benchmark, images of the same identity form sets. Forgenerative models, the standard way of dealing with sets is to represent eachas a one hot vector, and learn a conditional generative model$p(\mathbf{x}|\mathbf{y})$. This representation assumes that the number of setsis limited and known, such that the distribution over sets reduces to a simplemultinomial distribution. In contrast, we study a more generic problem wherethe number of sets is large and unknown. We introduce Set Distribution Networks(SDNs), a novel framework that learns to autoencode and freely generate sets.We achieve this by jointly learning a set encoder, set discriminator, setgenerator, and set prior. We show that SDNs are able to reconstruct image setsthat preserve salient attributes of the inputs in our benchmark datasets, andare also able to generate novel objects/identities. We examine the setsgenerated by SDN with a pre-trained 3D reconstruction network and a faceverification network, respectively, as a novel way to evaluate the quality ofgenerated sets of images.