Common Limitations of Image Processing Metrics: A Picture Story

  • 2021-04-12 17:03:42
  • Annika Reinke, Matthias Eisenmann, Minu D. Tizabi, Carole H. Sudre, Tim Rädsch, Michela Antonelli, Tal Arbel, Spyridon Bakas, M. Jorge Cardoso, Veronika Cheplygina, Keyvan Farahani, Ben Glocker, Doreen Heckmann-Nötzel, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Jens Kleesiek, Tahsin Kurc, Michal Kozubek, Bennett A. Landman, Geert Litjens, Klaus Maier-Hein, Bjoern Menze, Henning Müller, Jens Petersen, Mauricio Reyes, Nicola Rieke, Bram Stieltjes, Ronald M. Summers, Sotirios A. Tsaftaris, Bram van Ginneken, Annette Kopp-Schneider, Paul Jäger, Lena Maier-Hein
  • 192

Abstract

While the importance of automatic image analysis is increasing at an enormouspace, recent meta-research revealed major flaws with respect to algorithmvalidation. Specifically, performance metrics are key for objective,transparent and comparative performance assessment, but relatively littleattention has been given to the practical pitfalls when using specific metricsfor a given image analysis task. A common mission of several internationalinitiatives is therefore to provide researchers with guidelines and tools tochoose the performance metrics in a problem-aware manner. This dynamicallyupdated document has the purpose to illustrate important limitations ofperformance metrics commonly applied in the field of image analysis. Thecurrent version is based on a Delphi process on metrics conducted by aninternational consortium of image analysis experts.

 

Quick Read (beta)

loading the full paper ...