Abstract
In-context learning (ICL) is a critical emerging capability of large languagemodels (LLMs), enabling few-shot learning during inference by including a fewdemonstrations (demos) in the prompt. However, it has been found that ICL'sperformance can be sensitive to the choices of demos and their order. Thispaper investigates an unexplored new positional bias of ICL for the first time:we observe that the predictions and accuracy can drift drastically when thepositions of demos, the system prompt, and the user message in LLM input arevaried. We refer to this bias as DEMOS' POSITION IN PROMPT (DPP) bias. Wedesign a systematic evaluation pipeline to study this type of positional biasacross classification, question answering, summarization, and reasoning tasks.We introduce two metrics, ACCURACY-CHANGE and PREDICTION-CHANGE, to quantifynet gains and output volatility induced by changes in the demos' position.Extensive experiments on ten LLMs from four open-source model families (QWEN,LLAMA3, MISTRAL, COHERE) verify that the bias significantly affects theiraccuracy and predictions: placing demos at the start of the prompt yields themost stable and accurate outputs with gains of up to +6 points. In contrast,placing demos at the end of the user message flips over 30\% of predictionswithout improving correctness on QA tasks. Smaller models are most affected bythis sensitivity, though even large models remain marginally affected oncomplex tasks.