Abstract
Spurred by advancements in scale, large language models (LLMs) havedemonstrated strong few-shot learning ability via in-context learning (ICL).However, the performance of ICL has been shown to be highly sensitive to theselection of few-shot demonstrations. Selecting the most suitable examples ascontext remains an ongoing challenge and an open problem. Existing literaturehas highlighted the importance of selecting examples that are diverse orsemantically similar to the test sample while ignoring the fact that theoptimal selection dimension, i.e., diversity or similarity, is task-specific.Based on how the test sample is answered, we propose Iterative DemonstrationSelection (IDS) to leverage the merits of both dimensions. Using zero-shotchain-of-thought reasoning (Zero-shot-CoT), IDS iteratively selects examplesthat are diverse but still strongly correlated with the test sample as ICLdemonstrations. Specifically, IDS applies Zero-shot-CoT to the test samplebefore demonstration selection. The output reasoning path is then used tochoose demonstrations that are prepended to the test sample for inference. Thegenerated answer is followed by its corresponding reasoning path for extractinga new set of demonstrations in the next iteration. After several iterations,IDS adopts majority voting to obtain the final result. Through extensiveexperiments on tasks including reasoning, question answering, and topicclassification, we demonstrate that IDS can consistently outperform existingICL demonstration selection methods.