In this paper we investigate the feasibility of using synthetic data toaugment face datasets. In particular, we propose a novel generative adversarialnetwork (GAN) that can disentangle identity-related attributes fromnon-identity-related attributes. This is done by training an embedding networkthat maps discrete identity labels to an identity latent space that follows asimple prior distribution, and training a GAN conditioned on samples from thatdistribution. Our proposed GAN allows us to augment face datasets by generatingboth synthetic images of subjects in the training set and synthetic images ofnew subjects not in the training set. By using recent advances in GAN training,we show that the synthetic images generated by our model are photo-realistic,and that training with augmented datasets can indeed increase the accuracy offace recognition models as compared with models trained with real images alone.