Abstract
The Base-New Trade-off (BNT) problem universally exists during theoptimization of CLIP-based prompt tuning, where continuous fine-tuning on base(target) classes leads to a simultaneous decrease of generalization ability onnew (unseen) classes. Existing approaches attempt to regulate the prompt tuningprocess to balance BNT by appending constraints. However, imposed on the sametarget prompt, these constraints fail to fully avert the mutual exclusivitybetween the optimization directions for base and new. As a novel solution tothis challenge, we propose the plug-and-play Dual-Prompt Collaboration (DPC)framework, the first that decoupling the optimization processes of base and newtasks at the prompt level. Specifically, we clone a learnable parallel promptbased on the backbone prompt, and introduce a variable Weighting-Decouplingframework to independently control the optimization directions of dual promptsspecific to base or new tasks, thus avoiding the conflict in generalization.Meanwhile, we propose a Dynamic Hard Negative Optimizer, utilizing dual promptsto construct a more challenging optimization task on base classes forenhancement. For interpretability, we prove the feature channel invariance ofthe prompt vector during the optimization process, providing theoreticalsupport for the Weighting-Decoupling of DPC. Extensive experiments on multiplebackbones demonstrate that DPC can significantly improve base performancewithout introducing any external knowledge beyond the base classes, whilemaintaining generalization to new classes. Code is available at:https://github.com/JREion/DPC.