In this paper, we address the recent controversy between Lipschitzregularization and the choice of loss function for the training of GenerativeAdversarial Networks (GANs). One side argues that the success of the GANtraining should be attributed to the choice of loss function [16, 2, 5], whilethe other suggests that the Lipschitz regularization is the key to good results[17, 3, 18, 19]. We provide a theoretical and experimental analysis of howLipschitz regularization interacts with the loss function to derive thefollowing insights: (i) We show that popular GANs (NS-GAN , LS-GAN ,WGAN ) perform equally well when the discriminator is regularized with asmall Lipschitz constant, but the performance in terms of quality and diversitygets worse for larger Lipschitz constants, except for WGAN. (ii) We show thatall loss functions degenerate to linear ones for small Lipschitz constants toexplain why the performance of these GANs is similar. For higher Lipschitzconstants, we observe that only WGAN performs well while NS-GAN and LS-GANbreak down. For lower Lipschitz constants, NS-GAN and LS-GAN perform similarlyto WGAN only because they degenerate to the WGAN loss. In order to furtherillustrate this issue, we demonstrate that even "ridiculous" loss functionssuch as sin and exp have similar performance to NS-GAN, LS-GAN, and WGAN, whensmall Lipschitz constants are used.