Abstract
Popular PEFT methods reduce trainable parameter count for fine-tuning byparameterizing new low-rank or sparse trainable weights in parallel to thefrozen pre-trained weights $W$. However, these weights are trained fromscratch, and there exists a performance gap between these methods and fullfine-tuning, especially in low-budget settings. We introduce VectorFit, a newway of parameterization that efficiently utilizes the existing knowledgeembedded in $W$ by adaptively training their singular vectors and biases. Weshow that utilizing the structural and transformational properties of $W$ inthis way can lead to high-rank incremental weight matrices $\Delta W$,comparable to that of full fine-tuning. VectorFit delivers superior resultswith \textbf{9$\boldsymbol\times$} fewer trainable parameters than the leadingPEFT methods. Through comprehensive experiments across 19 datasets covering awide range of language and vision tasks such as natural language understandingand generation, question answering, image classification, and image generation,we demonstrate that VectorFit surpasses baselines in terms of performance as afunction of parameter-efficiency.