Abstract
Retrieval augmented generation (RAG) is a process where a large languagemodel (LLM) retrieves useful information from a database and then generates theresponses. It is becoming popular in enterprise settings for daily businessoperations. For example, Copilot for Microsoft 365 has accumulated millions ofbusinesses. However, the security implications of adopting such RAG-basedsystems are unclear. In this paper, we introduce ConfusedPilot, a class of securityvulnerabilities of RAG systems that confuse Copilot and cause integrity andconfidentiality violations in its responses. First, we investigate avulnerability that embeds malicious text in the modified prompt in RAG,corrupting the responses generated by the LLM. Second, we demonstrate avulnerability that leaks secret data, which leverages the caching mechanismduring retrieval. Third, we investigate how both vulnerabilities can beexploited to propagate misinformation within the enterprise and ultimatelyimpact its operations, such as sales and manufacturing. We also discuss theroot cause of these attacks by investigating the architecture of a RAG-basedsystem. This study highlights the security vulnerabilities in today's RAG-basedsystems and proposes design guidelines to secure future RAG-based systems.