Abstract
Multimodal Large Language Models (LLMs) are pivotal in revolutionizingcustomer support and operations by integrating multiple modalities such astext, images, and audio. Federated Prompt Learning (FPL) is a recently proposedapproach that combines pre-trained multimodal LLMs such as vision-languagemodels with federated learning to create personalized, privacy-preserving AIsystems. However, balancing the competing goals of personalization,generalization, and privacy remains a significant challenge.Over-personalization can lead to overfitting, reducing generalizability, whilestringent privacy measures, such as differential privacy, can hinder bothpersonalization and generalization. In this paper, we propose a DifferentiallyPrivate Federated Prompt Learning (DP-FPL) approach to tackle this challenge byleveraging a low-rank adaptation scheme to capture generalization whilemaintaining a residual term that preserves expressiveness for personalization.To ensure privacy, we introduce a novel method where we apply localdifferential privacy to the two low-rank components of the local prompt, andglobal differential privacy to the global prompt. Our approach mitigates theimpact of privacy noise on the model performance while balancing the tradeoffbetween personalization and generalization. Extensive experiments demonstratethe effectiveness of our approach over other benchmarks.