Abstract
We study model merging as a practical alternative to conventional adaptationstrategies for code-mixed NLP. Starting from a multilingual base model, we: (i)perform continued pre-training (CPT) on unlabeled code-mixed text to obtain anadapted checkpoint, (ii) merge checkpoint with the base model, and (iii)fine-tune (FT) on the downstream task data. We evaluate our approach forsentence classification (sentiment and hate speech) task in English-Hindi(En-Hi) and English-Spanish (En-Es) using XLM-R and Llama-3.2-1B models. Ourresults show that merged models consistently outperform full fine-tuning andCPT->FT. We observe gains of 2--5 points in F1 over full fine-tuning and ~1-2points over CPT->FT, indicating that unlabeled data is leveraged moreeffectively via merging than via CPT alone. Zero-/few-shot prompting withlarger LLMs (e.g., Llama-3.3-70B) lags behind fine-tuned and mergedcheckpoints, underscoring limits of in-context learning for code-mixed inputs.We further test cross-pair transfer by training on En-Hi and evaluating onEn-Ta and En-Ml: merged checkpoints transfer more strongly thanmonolingual-English baselines (e.g., TV/TIES variants reaching 0.65-0.68 F1 vs0.61-0.63 for full fine-tuning), suggesting that code-mixed knowledge is a morereliable substrate for low-resource pairs. We conclude with adaptation recipesmatched to common data regimes (labeled only; labeled+unlabeled; transfer-only)and discuss limitations and scaling considerations for broader tasks and largermodels.