Abstract
We present the NeurIPS 2021 consistency experiment, a larger-scale variant ofthe 2014 NeurIPS experiment in which 10% of conference submissions werereviewed by two independent committees to quantify the randomness in the reviewprocess. We observe that the two committees disagree on their accept/rejectrecommendations for 23% of the papers and that, consistent with the resultsfrom 2014, approximately half of the list of accepted papers would change ifthe review process were randomly rerun. Our analysis suggests that making theconference more selective would increase the arbitrariness of the process.Taken together with previous research, our results highlight the inherentdifficulty of objectively measuring the quality of research, and suggest thatauthors should not be excessively discouraged by rejected work.