Invariance to a broad array of image corruptions, such as warping, noise, orcolor shifts, is an important aspect of building robust models in computervision. Recently, several new data augmentations have been proposed thatsignificantly improve performance on ImageNet-C, a benchmark of suchcorruptions. However, there is still a lack of basic understanding on therelationship between data augmentations and test-time corruptions. To this end,we develop a feature space for image transforms, and then use a new measure inthis space between augmentations and corruptions called the Minimal SampleDistance to demonstrate there is a strong correlation between similarity andperformance. We then investigate recent data augmentations and observe asignificant degradation in corruption robustness when the test-time corruptionsare sampled to be perceptually dissimilar from ImageNet-C in this featurespace. Our results suggest that test error can be improved by training onperceptually similar augmentations, and data augmentations may not generalizewell beyond the existing benchmark. We hope our results and tools will allowfor more robust progress towards improving robustness to image corruptions.