Abstract
The well-documented presence of texture bias in modern convolutional neuralnetworks has led to a plethora of algorithms that promote an emphasis on shapecues, often to support generalization to new domains. Yet, common datasets,benchmarks and general model selection strategies are missing, and there is noagreed, rigorous evaluation protocol. In this paper, we investigatedifficulties and limitations when training networks with reduced texture bias.In particular, we also show that proper evaluation and meaningful comparisonsbetween methods are not trivial. We introduce BiasBed, a testbed for texture-and style-biased training, including multiple datasets and a range of existingalgorithms. It comes with an extensive evaluation protocol that includesrigorous hypothesis testing to gauge the significance of the results, despitethe considerable training instability of some style bias methods. Our extensiveexperiments, shed new light on the need for careful, statistically foundedevaluation protocols for style bias (and beyond). E.g., we find that somealgorithms proposed in the literature do not significantly mitigate the impactof style bias at all. With the release of BiasBed, we hope to foster a commonunderstanding of consistent and meaningful comparisons, and consequently fasterprogress towards learning methods free of texture bias. Code is available athttps://github.com/D1noFuzi/BiasBed