Abstract
Generative Adversarial Networks (GANs) can generate near photo realisticimages in narrow domains such as human faces. Yet, modeling complexdistributions of datasets such as ImageNet and COCO-Stuff remains challengingin unconditional settings. In this paper, we take inspiration from kerneldensity estimation techniques and introduce a non-parametric approach tomodeling distributions of complex datasets. We partition the data manifold intoa mixture of overlapping neighborhoods described by a datapoint and its nearestneighbors, and introduce a model, called instance-conditioned GAN (IC-GAN),which learns the distribution around each datapoint. Experimental results onImageNet and COCO-Stuff show that IC-GAN significantly improves overunconditional models and unsupervised data partitioning baselines. Moreover, weshow that IC-GAN can effortlessly transfer to datasets not seen during trainingby simply changing the conditioning instances, and still generate realisticimages. Finally, we extend IC-GAN to the class-conditional case and showsemantically controllable generation and competitive quantitative results onImageNet; while improving over BigGAN on ImageNet-LT. We will opensource ourcode and trained models to reproduce the reported results.