Privacy-Preserving Kickstarting Deep Reinforcement Learning with Privacy-Aware Learners

  • 2021-06-04 20:47:39
  • Parham Gohari, Bo Chen, Bo Wu, Matthew Hale, Ufuk Topcu
  • 0


Kickstarting deep reinforcement learning algorithms facilitate ateacher-student relationship among the agents and allow for a well-performingteacher to share demonstrations with a student to expedite the student'straining. However, despite the known benefits, the demonstrations may containsensitive information about the teacher's training data and existingkickstarting methods do not take any measures to protect it. Therefore, we usethe framework of differential privacy to develop a mechanism that securelyshares the teacher's demonstrations with the student. The mechanism allows forthe teacher to decide upon the accuracy of its demonstrations with respect tothe privacy budget that it consumes, thereby granting the teacher full controlover its data privacy. We then develop a kickstarted deep reinforcementlearning algorithm for the student that is privacy-aware because we calibrateits objective with the parameters of the teacher's privacy mechanism. Theprivacy-aware design of the algorithm makes it possible to kickstart thestudent's learning despite the perturbations induced by the privacy mechanism.From numerical experiments, we highlight three empirical results: (i) thealgorithm succeeds in expediting the student's learning, (ii) the studentconverges to a performance level that was not possible without thedemonstrations, and (iii) the student maintains its enhanced performance evenafter the teacher stops sharing useful demonstrations due to its privacy budgetconstraints.


Quick Read (beta)

loading the full paper ...