Abstract

Statistical hypothesis testing serves as statistical evidence for scientificinnovation. However, if the reported results are intentionally biased,hypothesis testing no longer controls the rate of false discovery. Inparticular, we study such selection bias in machine learning models where thereporter is motivated to promote an algorithmic innovation. When the number ofpossible configurations (e.g., datasets) is large, we show that the reportercan falsely report an innovation even if there is no improvement at all. Wepropose a `post-reporting' solution to this issue where the bias of thereported results is verified by another set of results. The theoreticalfindings are supported by experimental results with synthetic and real-worlddatasets.

 

Quick Read (beta)