Abstract
Large language models (LLMs) have shown impressive abilities in leveragingpretrained knowledge through prompting, but they often struggle with unseentasks, particularly in data-scarce scenarios. While cross-task in-contextlearning offers a direct solution for transferring knowledge across tasks, itstill faces critical challenges in terms of robustness, scalability, andefficiency. In this paper, we investigate whether cross-task transfer can beachieved via latent space steering without parameter updates or inputexpansion. Through an analysis of activation patterns in the latent space ofLLMs, we observe that the enhanced activations induced by in-context exampleshave consistent patterns across different tasks. Inspired by these findings, wepropose CAST, a novel Cross-task Activation Steering Transfer framework thatenables effective transfer by manipulating the model's internal activationstates. Our approach first selects influential and diverse samples fromhigh-resource tasks, then utilizes their contrastive representation-enhancedactivations to adapt LLMs to low-resource tasks. Extensive experiments acrossboth cross-domain and cross-lingual transfer settings show that our methodoutperforms competitive baselines and demonstrates superior scalability andlower computational costs.