Abstract
For many differentially private algorithms, such as the prominent noisystochastic gradient descent (DP-SGD), the analysis needed to bound the privacyleakage of a single training run is well understood. However, few studies havereasoned about the privacy leakage resulting from the multiple training runsneeded to fine tune the value of the training algorithm's hyperparameters. Inthis work, we first illustrate how simply setting hyperparameters based onnon-private training runs can leak private information. Motivated by thisobservation, we then provide privacy guarantees for hyperparameter searchprocedures within the framework of Renyi Differential Privacy. Our resultsimprove and extend the work of Liu and Talwar (STOC 2019). Our analysissupports our previous observation that tuning hyperparameters does indeed leakprivate information, but we prove that, under certain assumptions, this leakageis modest, as long as each candidate training run needed to selecthyperparameters is itself differentially private.