Rehabilitating the ColorChecker Dataset for Illuminant Estimation

  • 2018-09-17 16:53:27
  • Ghalia Hemrit, Graham D. Finlayson, Arjan Gijsenij, Peter Gehler, Simone Bianco, Brian Funt, Mark Drew, Lilong Shi
  • 2

Abstract

In a previous work, it was shown that there is a curious problem with thebenchmark ColorChecker dataset for illuminant estimation. To wit, this datasethas at least 3 different sets of ground-truths. Typically, for a singlealgorithm a single ground-truth is used. But then different algorithms, whoseperformance is measured with respect to different ground-truths, are comparedagainst each other and then ranked. This makes no sense. We show in this paperthat there are also errors in how each ground-truth set was calculated. As aresult, all performance rankings based on the ColorChecker dataset - and thereare scores of these - are inaccurate. In this paper, we re-generate a new 'recommended' set of ground-truth basedon the calculation methodology described by Shi and Funt. We then review theperformance evaluation of a range of illuminant estimation algorithms. Comparedwith the legacy ground-truths, we find that the difference in how algorithmsperform can be large, with many local rankings of algorithms being reversed. Finally, we draw the readers attention to our new 'open' data repositorywhich, we hope, will allow the ColorChecker set to be rehabilitated and onceagain to become a useful benchmark for illuminant estimation algorithms.

 

Quick Read (beta)

loading the full paper ...