Abstract
Supervised fine-tuning (SFT) is a pivotal approach to adapting large languagemodels (LLMs) for downstream tasks; however, performance often suffers from the``seesaw phenomenon'', where indiscriminate parameter updates yield progress oncertain tasks at the expense of others. To address this challenge, we propose anovel \emph{Core Parameter Isolation Fine-Tuning} (CPI-FT) framework.Specifically, we first independently fine-tune the LLM on each task to identifyits core parameter regions by quantifying parameter update magnitudes. Taskswith similar core regions are then grouped based on region overlap, formingclusters for joint modeling. We further introduce a parameter fusion technique:for each task, core parameters from its individually fine-tuned model aredirectly transplanted into a unified backbone, while non-core parameters fromdifferent tasks are smoothly integrated via Spherical Linear Interpolation(SLERP), mitigating destructive interference. A lightweight, pipelined SFTtraining phase using mixed-task data is subsequently employed, while freezingcore regions from prior tasks to prevent catastrophic forgetting. Extensiveexperiments on multiple public benchmarks demonstrate that our approachsignificantly alleviates task interference and forgetting, consistentlyoutperforming vanilla multi-task and multi-stage fine-tuning baselines.