Abstract
The performance of deep models, including Vision Transformers, is known to bevulnerable to adversarial attacks. Many existing defenses against theseattacks, such as adversarial training, rely on full-model fine-tuning to inducerobustness in the models. These defenses require storing a copy of the entiremodel, that can have billions of parameters, for each task. At the same time,parameter-efficient prompt tuning is used to adapt large transformer-basedmodels to downstream tasks without the need to save large copies. In thispaper, we examine parameter-efficient prompt tuning of Vision Transformers fordownstream tasks under the lens of robustness. We show that previousadversarial defense methods, when applied to the prompt tuning paradigm, sufferfrom gradient obfuscation and are vulnerable to adaptive attacks. We introduceADAPT, a novel framework for performing adaptive adversarial training in theprompt tuning paradigm. Our method achieves competitive robust accuracy of ~40%w.r.t. SOTA robustness methods using full-model fine-tuning, by tuning only ~1%of the number of parameters.