Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization

  • 2022-11-23 17:44:17
  • Tong Li, Jacob Nogas, Haochen Song, Harsh Kumar, Audrey Durand, Anna Rafferty, Nina Deliu, Sofia S. Villar, Joseph J. Williams
  • 0

Abstract

Multi-armed bandit algorithms like Thompson Sampling (TS) can be used toconduct adaptive experiments, in which maximizing reward means that data isused to progressively assign participants to more effective arms. Suchassignment strategies increase the risk of statistical hypothesis testsidentifying a difference between arms when there is not one, and failing toconclude there is a difference in arms when there truly is one. We tackle thisby introducing a novel heuristic algorithm, called TS-PostDiff (PosteriorProbability of Difference). TS-PostDiff takes a Bayesian approach to mixing TSand Uniform Random (UR): the probability a participant is assigned using URallocation is the posterior probability that the difference between two arms is'small' (below a certain threshold), allowing for more UR exploration whenthere is little or no reward to be gained. We evaluate TS-PostDiff againststate-of-the-art strategies. The empirical and simulation results helpcharacterize the trade-offs of these approaches between reward, False PositiveRate (FPR), and statistical power, as well as under which circumstances each iseffective. We quantify the advantage of TS-PostDiff in performing well acrossmultiple differences in arm means (effect sizes), showing the benefits ofadaptively changing randomization/exploration in TS in a "StatisticallyConsiderate" manner: reducing FPR and increasing statistical power whendifferences are small or zero and there is less reward to be gained, whileexploiting more when differences may be large. This highlights importantconsiderations for future algorithm development and analysis to better balancereward and statistical analysis.

 

Quick Read (beta)

loading the full paper ...