Abstract
Federated prompt learning benefits federated learning with CLIP-likeVision-Language Model's (VLM's) robust representation learning ability throughprompt learning. However, current federated prompt learning methods arehabitually restricted to the traditional FL paradigm, where the participatingclients are generally only allowed to download a single globally aggregatedmodel from the server. While justifiable for training full-sized models underfederated settings, in this work, we argue that this paradigm is ill-suited forlightweight prompts. By facilitating the clients to download multiplepre-aggregated prompts as fixed non-local experts, we propose PersonalizedFederated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework thatpersonalizes the prompt learning process through the lens of Mixture of Experts(MoE). pFedMoAP implements a local attention-based gating network that learnsto generate enhanced text features for better alignment with local image data,benefiting from both local and downloaded non-local adaptive prompt experts.Extensive experiments on 9 datasets under various federated settingsdemonstrate the efficacy of the proposed pFedMoAP algorithm. The code isavailable at https://github.com/ljaiverson/pFedMoAP.