Abstract
The rapid development of large language models (LLMs) in recent years haslargely focused on English, resulting in models that respond exclusively inEnglish. To adapt these models to other languages, continual pre-training (CP)is often employed, followed by supervised fine-tuning (SFT) to maintainconversational abilities. However, CP and SFT can reduce a model's ability tofilter harmful content. We propose Instruction Continual Pre-training (InsCP),which integrates instruction tags into the CP process to prevent loss ofconversational proficiency while acquiring new languages. Our experimentsdemonstrate that InsCP retains conversational and Reinforcement Learning fromHuman Feedback (RLHF) abilities. Empirical evaluations on language alignment,reliability, and knowledge benchmarks confirm the efficacy of InsCP. Notably,this approach requires only 0.1 billion tokens of high-qualityinstruction-following data, thereby reducing resource consumption.