Learning by contrasting positive and negative samples is a general strategyadopted by many methods. Noise contrastive estimation (NCE) for word embeddingsand translating embeddings for knowledge graphs are examples in NLP employingthis approach. In this work, we view contrastive learning as an abstraction ofall such methods and augment the negative sampler into a mixture distributioncontaining an adversarially learned sampler. The resulting adaptive samplerfinds harder negative examples, which forces the main model to learn a betterrepresentation of the data. We evaluate our proposal on learning wordembeddings, order embeddings and knowledge graph embeddings and observe bothfaster convergence and improved results on multiple metrics.