Abstract
Machine learning (ML) defenses protect against various risks to security,privacy, and fairness. Real-life models need simultaneous protection againstmultiple different risks which necessitates combining multiple defenses. Butcombining defenses with conflicting interactions in an ML model can beineffective, incurring a significant drop in the effectiveness of one or moredefenses being combined. Practitioners need a way to determine if a givencombination can be effective. Experimentally identifying effective combinationscan be time-consuming and expensive, particularly when multiple defenses needto be combined. We need an inexpensive, easy-to-use combination technique toidentify effective combinations. Ideally, a combination technique should be (a)accurate (correctly identifies whether a combination is effective or not), (b)scalable (allows combining multiple defenses), (c) non-invasive (requires nochange to the defenses being combined), and (d) general (is applicable todifferent types of defenses). Prior works have identified several ad-hoctechniques but none satisfy all the requirements above. We propose a principledcombination technique, Def\Con, to identify effective defense combinations.Def\Con meets all requirements, achieving 90% accuracy on eight combinationsexplored in prior work and 81% in 30 previously unexplored combinations that weempirically evaluate in this paper.