Abstract
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of largelanguage models by decomposing weight updates into low-rank matrices,significantly reducing storage and computational overhead. While effective,standard LoRA lacks mechanisms for uncertainty quantification, leading tooverconfident and poorly calibrated models. Bayesian variants of LoRA addressthis limitation, but at the cost of a significantly increased number oftrainable parameters, partially offsetting the original efficiency gains.Additionally, these models are harder to train and may suffer from unstableconvergence. In this work, we propose a novel parameter-efficient Bayesian LoRAvia subspace inference, demonstrating that effective uncertainty quantificationcan be achieved in very low-dimensional parameter spaces. The proposed methodachieves strong performance with improved calibration and generalization whilemaintaining computational efficiency. Our empirical findings show that, withthe appropriate projection of the weight space: (1) uncertainty can beeffectively modeled in a low-dimensional space, and (2) weight covariancesexhibit low ranks.