Abstract
We study the problem of in-context learning (ICL) with large language models(LLMs) on private datasets. This scenario poses privacy risks, as LLMs may leakor regurgitate the private examples demonstrated in the prompt. We propose anovel algorithm that generates synthetic few-shot demonstrations from theprivate dataset with formal differential privacy (DP) guarantees, and showempirically that it can achieve effective ICL. We conduct extensive experimentson standard benchmarks and compare our algorithm with non-private ICL andzero-shot solutions. Our results demonstrate that our algorithm can achievecompetitive performance with strong privacy levels. These results open up newpossibilities for ICL with privacy protection for a broad range ofapplications.