Abstract
In this paper, we introduce Nested Low-Rank Adaptation (NoRA), a novelapproach to parameter-efficient fine-tuning that extends the capabilities ofLow-Rank Adaptation (LoRA) techniques. Vanilla LoRA overlooks pre-trainedweight inheritance and still requires fine-tuning numerous parameters. Toaddresses these issues, our NoRA adopts a dual-layer nested structure withSingular Value Decomposition (SVD), effectively leveraging original matrixknowledge while reducing tunable parameters. Specifically, NoRA freezes theouter LoRA weights and utilizes an inner LoRA design, providing enhancedcontrol over model optimization. This approach allows the model to moreprecisely adapt to specific tasks while maintaining a compact parameter space.By freezing outer LoRA weights and using an inner LoRA design, NoRA enablesprecise task adaptation with a compact parameter space. Evaluations on tasksincluding commonsense reasoning with large language models, fine-tuningvision-language models, and subject-driven generation demonstrate NoRA'ssuperiority over LoRA and its variants. Code will be released upon acceptance.