Abstract
Prompt learning has emerged as an efficient alternative for fine-tuningfoundational models, such as CLIP, for various downstream tasks. Conventionallytrained using the task-specific objective, i.e., cross-entropy loss, promptstend to overfit downstream data distributions and find it challenging tocapture task-agnostic general features from the frozen CLIP. This leads to theloss of the model's original generalization capability. To address this issue,our work introduces a self-regularization framework for prompting calledPromptSRC (Prompting with Self-regulating Constraints). PromptSRC guides theprompts to optimize for both task-specific and task-agnostic generalrepresentations using a three-pronged approach by: (a) regulating promptedrepresentations via mutual agreement maximization with the frozen model, (b)regulating with self-ensemble of prompts over the training trajectory to encodetheir complementary strengths, and (c) regulating with textual diversity tomitigate sample diversity imbalance with the visual branch. To the best of ourknowledge, this is the first regularization framework for prompt learning thatavoids overfitting by jointly attending to pre-trained model features, thetraining trajectory during prompting, and the textual diversity. PromptSRCexplicitly steers the prompts to learn a representation space that maximizesperformance on downstream tasks without compromising CLIP generalization. Weperform extensive experiments on 4 benchmarks where PromptSRC overall performsfavorably well compared to the existing methods. Our code and pre-trainedmodels are publicly available at: https://github.com/muzairkhattak/PromptSRC.