Abstract
Recent advancements in Web AI agents have demonstrated remarkablecapabilities in addressing complex web navigation tasks. However, emergingresearch shows that these agents exhibit greater vulnerability compared tostandalone Large Language Models (LLMs), despite both being built upon the samesafety-aligned models. This discrepancy is particularly concerning given thegreater flexibility of Web AI Agent compared to standalone LLMs, which mayexpose them to a wider range of adversarial user inputs. To build a scaffoldthat addresses these concerns, this study investigates the underlying factorsthat contribute to the increased vulnerability of Web AI agents. Notably, thisdisparity stems from the multifaceted differences between Web AI agents andstandalone LLMs, as well as the complex signals - nuances that simpleevaluation metrics, such as success rate, often fail to capture. To tacklethese challenges, we propose a component-level analysis and a more granular,systematic evaluation framework. Through this fine-grained investigation, weidentify three critical factors that amplify the vulnerability of Web AIagents; (1) embedding user goals into the system prompt, (2) multi-step actiongeneration, and (3) observational capabilities. Our findings highlights thepressing need to enhance security and robustness in AI agent design and provideactionable insights for targeted defense strategies.