Abstract
With the rapid growth of deep learning, there is an increasing availabilityof open-source models for various tasks. However, single fine-tuned modelsoften fall short of meeting the diverse needs of users. Model merging has thusemerged as an efficient method to integrate the capabilities of existing modelsinto a unified model. Nevertheless, existing model merging methods facechallenging trade-offs between performance and deployment costs, primarily dueto task interference. For the first time, we reveal that task interference isevident in the frequency domain of model parameters, yet current efforts onlyfocus on spatial domain solutions, which are largely ineffective in addressingfrequency domain interference. To mitigate the impact of frequency domaininterference, we propose FR-Merging, an innovative method that effectivelyfilters harmful frequency domain interference on the backbone with minimalcomputational overhead. Since performance loss is inevitable with cost-freemethods, we propose a lightweight task-specific expert module that dynamicallycompensates for information loss during merging. This proposed framework,FREE-Merging (FR-Merging with experts), strikes a balanced trade-off betweentraining cost, inference latency, storage requirements, and performance. Wedemonstrate the effectiveness of both FR-Merging and FREE-Merging on multipletasks across CV, NLP, and Multi-Modal domains and show that they can beflexibly adapted to specific needs.