Abstract
Multilingual neural machine translation models support fine-tuning hundredsof languages simultaneously. However, fine-tuning on full parameters solely isinefficient potentially leading to negative interactions among languages. Inthis work, we demonstrate that the fine-tuning for a language occurs in itsintrinsic language-specific subspace with a tiny fraction of entire parameters.Thus, we propose language-specific LoRA to isolate intrinsic language-specificsubspaces. Furthermore, we propose architecture learning techniques andintroduce a gradual pruning schedule during fine-tuning to exhaustively explorethe optimal setting and the minimal intrinsic subspaces for each language,resulting in a lightweight yet effective fine-tuning procedure. Theexperimental results on a 12-language subset and a 30-language subset ofFLORES-101 show that our methods not only outperform full-parameter fine-tuningup to 2.25 spBLEU scores but also reduce trainable parameters to $0.4\%$ forhigh and medium-resource languages and $1.6\%$ for low-resource ones.