Taming GANs with Lookahead

  • 2020-06-25 17:13:23
  • Tatjana Chavdarova, Matteo Pagliardini, Martin Jaggi, Francois Fleuret
  • 65

Abstract

Generative Adversarial Networks are notoriously challenging to train. Theunderlying minimax optimization is highly susceptible to the variance of thestochastic gradient and the rotational component of the associated game vectorfield. We empirically demonstrate the effectiveness of the Lookaheadmeta-optimization method for optimizing games, originally proposed for standardminimization. The backtracking step of Lookahead naturally handles therotational game dynamics, which in turn enables the gradient ascent descentmethod to converge on challenging toy games often analyzed in the literature.Moreover, it implicitly handles high variance without using large mini-batches,known to be essential for reaching state of the art performance. Experimentalresults on MNIST, SVHN, and CIFAR-10, demonstrate a clear advantage ofcombining Lookahead with Adam or extragradient, in terms of performance, memoryfootprint, and improved stability. Using 30-fold fewer parameters and 16-foldsmaller minibatches we outperform the reported performance of theclass-dependent BigGAN on CIFAR-10 by obtaining FID of $13.65$ \emph{without}using the class labels, bringing state-of-the-art GAN training within reach ofcommon computational resources.

 

Quick Read (beta)

loading the full paper ...