Abstract
This paper presents adaptive conformal selection (ACS), an interactiveframework for model-free selection with guaranteed error control. Building onconformal selection (Jin and Cand\`es, 2023b), ACS generalizes the approach tosupport human-in-the-loop adaptive data analysis. Under the ACS framework, wecan partially reuse the data to boost the selection power, make decisions onthe fly while exploring the data, and incorporate new information orpreferences as they arise. The key to ACS is a carefully designed principlethat controls the information available for decision making, allowing the dataanalyst to explore the data adaptively while maintaining rigorous control ofthe false discovery rate (FDR). Based on the ACS framework, we provide concreteselection algorithms for various goals, including model update/selection,diversified selection, and incorporating newly available labeled data. Theeffectiveness of ACS is demonstrated through extensive numerical simulationsand real-data applications in large language model (LLM) deployment and drugdiscovery.