Abstract

Consistently checking the statistical significance of experimental results isone of the mandatory methodological steps to address the so-called"reproducibility crisis" in deep reinforcement learning. In this tutorialpaper, we explain how to determine the number of random seeds one should use toprovide a statistically significant comparison of the performance of twoalgorithms. We also discuss the influence of deviations from the assumptionsusually made by statistical tests, we provide guidelines to counter theirnegative effects and some code to perform the tests.

 

Quick Read (beta)