Abstract
LLM-integrated app systems extend the utility of Large Language Models (LLMs)with third-party apps that are invoked by a system LLM using interleavedplanning and execution phases to answer user queries. These systems introducenew attack vectors where malicious apps can cause integrity violation ofplanning or execution, availability breakdown, or privacy compromise duringexecution. In this work, we identify new attacks impacting the integrity of planning, aswell as the integrity and availability of execution in LLM-integrated apps, anddemonstrate them against IsolateGPT, a recent solution designed to mitigateattacks from malicious apps. We propose Abstract-Concrete-Execute (ACE), a newsecure architecture for LLM-integrated app systems that provides securityguarantees for system planning and execution. Specifically, ACE decouplesplanning into two phases by first creating an abstract execution plan usingonly trusted information, and then mapping the abstract plan to a concrete planusing installed system apps. We verify that the plans generated by our systemsatisfy user-specified secure information flow constraints via static analysison the structured plan output. During execution, ACE enforces data andcapability barriers between apps, and ensures that the execution is conductedaccording to the trusted abstract plan. We show experimentally that our systemis secure against attacks from the INJECAGENT benchmark, a standard benchmarkfor control flow integrity in the face of indirect prompt injection attacks,and our newly introduced attacks. Our architecture represents a significantadvancement towards hardening LLM-based systems containing system facilities ofvarying levels of trustworthiness.