Abstract
Multi-task large language models (MTLLMs) are important for many applicationsat the wireless edge, where users demand specialized models to handle multipletasks efficiently. However, training MTLLMs is complex and exhaustive,particularly when tasks are subject to change. Recently, the concept of modelfusion via task vectors has emerged as an efficient approach for combiningfine-tuning parameters to produce an MTLLM. In this paper, the problem ofenabling edge users to collaboratively craft such MTTLMs via tasks vectors isstudied, under the assumption of worst-case adversarial attacks. To this end,first the influence of adversarial noise to multi-task model fusion isinvestigated and a relationship between the so-called weight disentanglementerror and the mean squared error (MSE) is derived. Using hypothesis testing, itis directly shown that the MSE increases interference between task vectors,thereby rendering model fusion ineffective. Then, a novel resilient MTLLMfusion (R-MTLLMF) is proposed, which leverages insights about the LLMarchitecture and fine-tuning process to safeguard task vector aggregation underadversarial noise by realigning the MTLLM. The proposed R-MTLLMF is thencompared for both worst-case and ideal transmission scenarios to study theimpact of the wireless channel. Extensive model fusion experiments with visionLLMs demonstrate R-MTLLMF's effectiveness, achieving close-to-baselineperformance across eight different tasks in ideal noise scenarios andsignificantly outperforming unprotected model fusion in worst-case scenarios.The results further advocate for additional physical layer protection for aholistic approach to resilience, from both a wireless and LLM perspective.