Abstract
We propose an adaptive sampling approach for multiple testing which aims tomaximize statistical power while ensuring anytime false discovery control. Weconsider $n$ distributions whose means are partitioned by whether they arebelow or equal to a baseline (nulls), versus above the baseline (actualpositives). In addition, each distribution can be sequentially and repeatedlysampled. Inspired by the multi-armed bandit literature, we provide an algorithmthat takes as few samples as possible to exceed a target true positiveproportion (i.e. proportion of actual positives discovered) while givinganytime control of the false discovery proportion (nulls predicted as actualpositives). Our sample complexity results match known information theoreticlower bounds and through simulations we show a substantial performanceimprovement over uniform sampling and an adaptive elimination style algorithm.Given the simplicity of the approach, and its sample efficiency, the method haspromise for wide adoption in the biological sciences, clinical testing for drugdiscovery, and online A/B/n testing problems.