Abstract
Task vectors offer a compelling mechanism for accelerating inference inin-context learning (ICL) by distilling task-specific information into asingle, reusable representation. Despite their empirical success, theunderlying principles governing their emergence and functionality remainunclear. This work proposes the Linear Combination Conjecture, positing thattask vectors act as single in-context demonstrations formed through linearcombinations of the original ones. We provide both theoretical and empiricalsupport for this conjecture. First, we show that task vectors naturally emergein linear transformers trained on triplet-formatted prompts through losslandscape analysis. Next, we predict the failure of task vectors onrepresenting high-rank mappings and confirm this on practical LLMs. Ourfindings are further validated through saliency analyses and parametervisualization, suggesting an enhancement of task vectors by injecting multipleones into few-shot prompts. Together, our results advance the understanding oftask vectors and shed light on the mechanisms underlying ICL intransformer-based models.