Abstract
Large Language Model (LLM) agents take irreversible real-world actions, yet no system gives formally grounded, task-level safety guarantees for their plans. We examined this gap in a PRISMA 2020 systematic review of six databases (20-29 May 2026) covering LLM agent specification, verification, and enforcement (2022-2026). We narratively synthesized 38 studies, assessed risk of bias per study, and graded evidence certainty with GRADE. First, specification is the bottleneck: natural-language-to-formal translation reaches only 24.8-34.2% semantic correctness on one Planning Domain Definition Language (PDDL) benchmark. Second, runtime monitoring is the most mature enforcement strategy, reducing unsafe actions by 65.37% in one household-agent study, without complete guarantees. Third, in one $τ$-Bench study of transactional tool-use agents, blocking 94% of unsafe actions left safe task completion below 5%, as agents found unsafe alternatives (the verifier tax). Fourth, no approach simultaneously achieves soundness, completeness, scalability, semantic grounding, and utility preservation. We contribute a three-level taxonomy and a ten-problem research agenda.