Pre-trained language models have proven their unique powers in capturingimplicit language features. However, most pre-training approaches focus on theword-level training objective, while sentence-level objectives are rarelystudied. In this paper, we propose Contrastive LEArning for sentenceRepresentation (CLEAR), which employs multiple sentence-level augmentationstrategies in order to learn a noise-invariant sentence representation. Theseaugmentations include word and span deletion, reordering, and substitution.Furthermore, we investigate the key reasons that make contrastive learningeffective through numerous experiments. We observe that different sentenceaugmentations during pre-training lead to different performance improvements onvarious downstream tasks. Our approach is shown to outperform multiple existingmethods on both SentEval and GLUE benchmarks.