Abstract
Continual learning in large language models (LLMs) is prone to catastrophicforgetting, where adapting to new tasks significantly degrades performance onpreviously learned ones. Existing methods typically rely on low-rank,parameter-efficient updates that limit the model's expressivity and introduceadditional parameters per task, leading to scalability issues. To address theselimitations, we propose a novel continual full fine-tuning approach leveragingadaptive singular value decomposition (SVD). Our method dynamically identifiestask-specific low-rank parameter subspaces and constrains updates to beorthogonal to critical directions associated with prior tasks, thus effectivelyminimizing interference without additional parameter overhead or storingprevious task gradients. We evaluate our approach extensively on standardcontinual learning benchmarks using both encoder-decoder (T5-Large) anddecoder-only (LLaMA-2 7B) models, spanning diverse tasks includingclassification, generation, and reasoning. Empirically, our method achievesstate-of-the-art results, up to 7% higher average accuracy than recentbaselines like O-LoRA, and notably maintains the model's general linguisticcapabilities, instruction-following accuracy, and safety throughout thecontinual learning process by reducing forgetting to near-negligible levels.Our adaptive SVD framework effectively balances model plasticity and knowledgeretention, providing a practical, theoretically grounded, and computationallyscalable solution for continual learning scenarios in large language models.