Abstract
Transfer learning is a standard technique to improve performance on taskswith limited data. However, for medical imaging, the value of transfer learningis less clear. This is likely due to the large domain mismatch between theusual natural-image pre-training (e.g. ImageNet) and medical images. However,recent advances in transfer learning have shown substantial improvements fromscale. We investigate whether modern methods can change the fortune of transferlearning for medical imaging. For this, we study the class of large-scalepre-trained networks presented by Kolesnikov et al. on three diverse imagingtasks: chest radiography, mammography, and dermatology. We study both transferperformance and critical properties for the deployment in the medical domain,including: out-of-distribution generalization, data-efficiency, sub-groupfairness, and uncertainty estimation. Interestingly, we find that for some ofthese properties transfer from natural to medical images is indeed extremelyeffective, but only when performed at sufficient scale.