Abstract
Large Language Models (LLMs) are driving advancements in artificialintelligence by increasing the scale of model parameters, which hassignificantly enhanced generalization ability and unlocked new capabilities inpractice. However, their performance in specific downstream tasks is usuallyhindered by their knowledge boundaries on these tasks. Thus, fine-tuningtechniques, especially the widely used Low-Rank Adaptation (LoRA) method, havebeen introduced to expand the boundaries on these tasks, whereas LoRA wouldunderperform on certain tasks owing to its potential overfitting on thesetasks. To overcome this overfitting and improve the performance of LoRA, wepropose the flexible low rank adaptation (Flexora) method to automatically andflexibly select the most important layers needing to be fine-tuned to achievethe best performance on different downstream tasks. Specifically, Flexorafirstly frames this layer selection problem as a well-defined hyperparameteroptimization (HPO) problem, then addresses it using the unrolleddifferentiation (UD) method, and finally selects the most useful layers basedon the optimized hyperparameters. Our extensive experiments on many pretrainedmodels and natural language tasks show that Flexora is able to consistentlyimprove over the existing baselines, indicating the effectiveness of ourFlexora in practice. We additionally provide insightful theoretical results andmany ablation studies to deliver a comprehensive understanding of our Flexora.