Abstract
Large Language Models (LLMs) have become indispensable in numerous real-worldapplications. Unfortunately, fine-tuning these models at scale, especially infederated settings where data privacy and communication efficiency arecritical, presents significant challenges. Existing methods often resort toparameter-efficient fine-tuning (PEFT) to mitigate communication overhead, butthis typically comes at the cost of model accuracy. To address theselimitations, we propose federated full-parameter tuning at scale for LLMs(Ferret), the first first-order method with shared randomness to enablescalable full-parameter tuning of LLMs across decentralized data sources whilemaintaining competitive model accuracy. Ferret accomplishes this through threeaspects: (1) it employs widely applied first-order methods for efficient localupdates; (2) it projects these updates into a low-dimensional space toconsiderably reduce communication overhead; and (3) it reconstructs localupdates from this low-dimensional space with shared randomness to facilitateeffective full-parameter global aggregation, ensuring fast convergence andcompetitive final performance. Our rigorous theoretical analyses and insightsalong with extensive experiments, show that Ferret significantly enhances thescalability of existing federated full-parameter tuning approaches by achievinghigh computational efficiency, reduced communication overhead, and fastconvergence, all while maintaining competitive model accuracy. Ourimplementation is available at https://github.com/allen4747/Ferret.