Learning data representations that are useful for various downstream tasks isa cornerstone of artificial intelligence. While existing methods are typicallyevaluated on downstream tasks such as classification or generative imagequality, we propose to assess representations through their usefulness indownstream control tasks, such as reaching or pushing objects. By training over10,000 reinforcement learning policies, we extensively evaluate to what extentdifferent representation properties affect out-of-distribution (OOD)generalization. Finally, we demonstrate zero-shot transfer of these policiesfrom simulation to the real world, without any domain randomization orfine-tuning. This paper aims to establish the first systematic characterizationof the usefulness of learned representations for real-world OOD downstreamtasks.