Remote sensing and automatic earth monitoring are key to solve global-scalechallenges such as disaster prevention, land use monitoring, or tacklingclimate change. Although there exist vast amounts of remote sensing data, mostof it remains unlabeled and thus inaccessible for supervised learningalgorithms. Transfer learning approaches can reduce the data requirements ofdeep learning algorithms. However, most of these methods are pre-trained onImageNet and their generalization to remote sensing imagery is not guaranteeddue to the domain gap. In this work, we propose Seasonal Contrast (SeCo), aneffective pipeline to leverage unlabeled data for in-domain pre-training ofremote sensing representations. The SeCo pipeline is composed of two parts.First, a principled procedure to gather large-scale, unlabeled and uncuratedremote sensing datasets containing images from multiple Earth locations atdifferent timestamps. Second, a self-supervised algorithm that takes advantageof time and position invariance to learn transferable representations forremote sensing applications. We empirically show that models trained with SeCoachieve better performance than their ImageNet pre-trained counterparts andstate-of-the-art self-supervised learning methods on multiple downstream tasks.The datasets and models in SeCo will be made public to facilitate transferlearning and enable rapid progress in remote sensing applications.