Abstract
Recent advances in operating system (OS) agents have enabled vision-languagemodels (VLMs) to directly control a user's computer. Unlike conventional VLMsthat passively output text, OS agents autonomously perform computer-based tasksin response to a single user prompt. OS agents do so by capturing, parsing, andanalysing screenshots and executing low-level actions via applicationprogramming interfaces (APIs), such as mouse clicks and keyboard inputs. Thisdirect interaction with the OS significantly raises the stakes, as failures ormanipulations can have immediate and tangible consequences. In this work, weuncover a novel attack vector against these OS agents: Malicious Image Patches(MIPs), adversarially perturbed screen regions that, when captured by an OSagent, induce it to perform harmful actions by exploiting specific APIs. Forinstance, a MIP can be embedded in a desktop wallpaper or shared on socialmedia to cause an OS agent to exfiltrate sensitive user data. We show that MIPsgeneralise across user prompts and screen configurations, and that they canhijack multiple OS agents even during the execution of benign instructions.These findings expose critical security vulnerabilities in OS agents that haveto be carefully addressed before their widespread deployment.