Abstract
The widespread availability of off-the-shelf machine learning models poses achallenge: which model, of the many available candidates, should be chosen fora given data analysis task? This question of model selection is traditionallyanswered by collecting and annotating a validation dataset -- a costly andtime-intensive process. We propose a method for active model selection, usingpredictions from candidate models to prioritize the labeling of test datapoints that efficiently differentiate the best candidate. Our method, CODA,performs consensus-driven active model selection by modeling relationshipsbetween classifiers, categories, and data points within a probabilisticframework. The framework uses the consensus and disagreement between models inthe candidate pool to guide the label acquisition process, and Bayesianinference to update beliefs about which model is best as more information iscollected. We validate our approach by curating a collection of 26 benchmarktasks capturing a range of model selection scenarios. CODA outperforms existingmethods for active model selection significantly, reducing the annotationeffort required to discover the best model by upwards of 70% compared to theprevious state-of-the-art. Code and data are available athttps://github.com/justinkay/coda.