Abstract
Federated learning has become a promising solution for collaboration amongmedical institutions. However, data owned by each institution would be highlyheterogeneous and the distribution is always non-independent and identicaldistribution (non-IID), resulting in client drift and unsatisfactoryperformance. Despite existing federated learning methods attempting to solvethe non-IID problems, they still show marginal advantages but rely on frequentcommunication which would incur high costs and privacy concerns. In this paper,we propose a novel federated learning method: \textbf{Fed}erated learning via\textbf{V}aluable \textbf{C}ondensed \textbf{K}nowledge (FedVCK). We enhancethe quality of condensed knowledge and select the most necessary knowledgeguided by models, to tackle the non-IID problem within limited communicationbudgets effectively. Specifically, on the client side, we condense theknowledge of each client into a small dataset and further enhance thecondensation procedure with latent distribution constraints, facilitating theeffective capture of high-quality knowledge. During each round, we specificallytarget and condense knowledge that has not been assimilated by the currentmodel, thereby preventing unnecessary repetition of homogeneous knowledge andminimizing the frequency of communications required. On the server side, wepropose relational supervised contrastive learning to provide more supervisionsignals to aid the global model updating. Comprehensive experiments acrossvarious medical tasks show that FedVCK can outperform state-of-the-art methods,demonstrating that it's non-IID robust and communication-efficient.