Abstract
The rise of large language models (LLMs) has introduced new privacychallenges, particularly during inference where sensitive information inprompts may be exposed to proprietary LLM APIs. In this paper, we address theproblem of formally protecting the sensitive information contained in a promptwhile maintaining response quality. To this end, first, we introduce acryptographically inspired notion of a prompt sanitizer which transforms aninput prompt to protect its sensitive tokens. Second, we proposePr$\epsilon\epsilon$mpt, a novel system that implements a prompt sanitizer.Pr$\epsilon\epsilon$mpt categorizes sensitive tokens into two types: (1) thosewhere the LLM's response depends solely on the format (such as SSNs, creditcard numbers), for which we use format-preserving encryption (FPE); and (2)those where the response depends on specific values, (such as age, salary) forwhich we apply metric differential privacy (mDP). Our evaluation demonstratesthat Pr$\epsilon\epsilon$mpt is a practical method to achieve meaningfulprivacy guarantees, while maintaining high utility compared to unsanitizedprompts, and outperforming prior methods