Abstract
Supervised fine-tuning (SFT) is a crucial step for large language models(LLMs), enabling them to align with human instructions and enhance theircapabilities in downstream tasks. When the models are required to align with abroader range of downstream tasks, or there is a desire to notably improve theperformance on a specific task, a substantial increase in fine-tuning dataoften emerges as the solution. However, we find that large-scale increases ininstruction data can disrupt the world knowledge previously stored in the LLMs,i.e., world knowledge forgetting. In this paper, we introduce LoRAMoE toaddress the above challenge. The LoRAMoE is a plugin version of Mixture ofExperts (MoE). The plugin form ensures the integrity of world knowledge byfreezing the backbone model during the training phase. We then propose the useof localized balancing constraints to coordinate parts of experts for taskutilization, meanwhile enabling other experts to fully leverage the worldknowledge stored in the models. Experimental results demonstrate that LoRAMoEcan reasonably coordinate experts based on data type during inference, and evendramatically increasing instruction data does not result in knowledgeforgetting. Moreover, LoRAMoE provides additional benefits for the performanceof downstream tasks, indicating the potential of our approach for multi-tasklearning.