Abstract
Crowdworking is a cost-efficient solution for acquiring class labels. Sincethese labels are subject to noise, various approaches to learning from crowdshave been proposed. Typically, these approaches are evaluated with defaulthyperparameter configurations, resulting in unfair and suboptimal performance,or with hyperparameter configurations tuned via a validation set with groundtruth class labels, representing an often unrealistic scenario. Moreover, bothsetups can produce different approach rankings, complicating study comparisons.Therefore, we introduce crowd-hpo as a framework for evaluating approaches tolearning from crowds in combination with criteria to select well-performinghyperparameter configurations with access only to noisy crowd-labeledvalidation data. Extensive experiments with neural networks demonstrate thatthese criteria select hyperparameter configurations, which improve the learningfrom crowd approaches' generalization performances, measured on separate testsets with ground truth labels. Hence, incorporating such criteria intoexperimental studies is essential for enabling fairer and more realisticbenchmarking.