Abstract
We introduce adaptive learn-then-test (aLTT), an efficient hyperparameterselection procedure that provides finite-sample statistical guarantees on thepopulation risk of AI models. Unlike the existing learn-then-test (LTT)technique, which relies on conventional p-value-based multiple hypothesistesting (MHT), aLTT implements sequential data-dependent MHT with earlytermination by leveraging e-processes. As a result, aLTT can reduce the numberof testing rounds, making it particularly well-suited for scenarios in whichtesting is costly or presents safety risks. Apart from maintaining statisticalvalidity, in applications such as online policy selection for offlinereinforcement learning and prompt engineering, aLTT is shown to achieve thesame performance as LTT while requiring only a fraction of the testing rounds.