Abstract
Federated Learning (FL) aims to train models across decentralized clients ordevices holding local data without the need for centralized data collection,thus enhancing data privacy and security. However, achieving bothgeneralization and personalization in heterogeneous settings remains asignificant challenge. To address this, we introduce FedOT, a novel approachthat leverages black-box foundation models. FedOT shares only a globaltask-dependent classifier across clients while locally adapting featuresthrough orthogonal transformations. By enforcing orthogonality, FedOT mitigatesgradient conflicts across diverse clients, preserves semantic integrity, andachieves robust performance even in the presence of substantial dataheterogeneity. The strategy of combining global and local parameters enables amore balanced approach for both generalization and personalization,outperforming baseline FL methods across multiple benchmarks. Furthermore, ourextensive analysis confirms that joint optimization of global classifiers andlocal orthogonal transformations yields superior performance and suggestsbroader applicability.