Abstract
Autonomous agents that execute human tasks by controlling computers canenhance human productivity and application accessibility. However, progress inthis field will be driven by realistic and reproducible benchmarks. We presentAndroidWorld, a fully functional Android environment that provides rewardsignals for 116 programmatic tasks across 20 real-world Android apps. Unlikeexisting interactive environments, which provide a static test set,AndroidWorld dynamically constructs tasks that are parameterized and expressedin natural language in unlimited ways, thus enabling testing on a much largerand more realistic suite of tasks. Reward signals are derived from thecomputer's system state, making them durable across task variations andextensible across different apps. To demonstrate AndroidWorld's benefits andmode of operation, we introduce a new computer control agent, M3A. M3A cancomplete 30.6% of the AndroidWorld's tasks, leaving ample room for future work.Furthermore, we adapt a popular desktop web agent to work on Android, which wefind to be less effective on mobile, suggesting future research is needed toachieve universal, cross-domain agents. Finally, we conduct a robustnessanalysis by testing M3A against a range of task variations on a representativesubset of tasks, demonstrating that variations in task parameters cansignificantly alter a task's complexity and, consequently, an agent'sperformance, highlighting the importance of testing agents under diverseconditions. AndroidWorld and the experiments in this paper are available athttps://github.com/google-research/android_world.