Abstract
We introduce a simple (one line of code) modification to the GenerativeAdversarial Network (GAN) training algorithm that materially improves resultswith no increase in computational cost: When updating the generator parameters,we simply zero out the gradient contributions from the elements of the batchthat the critic scores as `least realistic'. Through experiments on manydifferent GAN variants, we show that this `top-k update' procedure is agenerally applicable improvement. In order to understand the nature of theimprovement, we conduct extensive analysis on a simple mixture-of-Gaussiansdataset and discover several interesting phenomena. Among these is that, whengradient updates are computed using the worst-scoring batch elements, samplescan actually be pushed further away from their nearest mode. We also apply ourmethod to recent GAN variants and improve state-of-the-art FID for conditionalgeneration from 9.21 to 8.57 on CIFAR-10.