Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions

  • 2021-04-08 15:15:22
  • Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, Nam Vo, Peggy Bui, Samantha Winter, Patricia MacWilliams, Greg S. Corrado, Umesh Telang, Yun Liu, Taylan Cemgil, Alan Karthikesalingam, Balaji Lakshminarayanan, Jim Winkens
  • 31

Abstract

We develop and rigorously evaluate a deep learning based system that canaccurately classify skin conditions while detecting rare conditions for whichthere is not enough data available for training a confident classifier. Weframe this task as an out-of-distribution (OOD) detection problem. Our novelapproach, hierarchical outlier detection (HOD) assigns multiple abstentionclasses for each training outlier class and jointly performs a coarseclassification of inliers vs. outliers, along with fine-grained classificationof the individual classes. We demonstrate the effectiveness of the HOD loss inconjunction with modern representation learning approaches (BiT, SimCLR, MICLe)and explore different ensembling strategies for further improving the results.We perform an extensive subgroup analysis over conditions of varying risklevels and different skin types to investigate how the OOD detectionperformance changes over each subgroup and demonstrate the gains of ourframework in comparison to baselines. Finally, we introduce a cost metric toapproximate downstream clinical impact. We use this cost metric to compare theproposed method against a baseline system, thereby making a stronger case forthe overall system effectiveness in a real-world deployment scenario.

 

Quick Read (beta)

loading the full paper ...