Abstract
Creating autonomous virtual agents capable of using arbitrary software on anydigital device remains a major challenge for artificial intelligence. Two keyobstacles hinder progress: insufficient infrastructure for building virtualagents in real-world environments, and the need for in-the-wild evaluation offundamental agent abilities. To address this, we introduce AgentStudio, anonline, realistic, and multimodal toolkit that covers the entire lifecycle ofagent development. This includes environment setups, data collection, agentevaluation, and visualization. The observation and action spaces are highlygeneric, supporting both function calling and human-computer interfaces. Thisversatility is further enhanced by AgentStudio's graphical user interfaces,which allow efficient development of datasets and benchmarks in real-worldsettings. To illustrate, we introduce a visual grounding dataset and areal-world benchmark suite, both created with our graphical interfaces.Furthermore, we present several actionable insights derived from AgentStudio,e.g., general visual grounding, open-ended tool creation, learning from videos,etc. We have open-sourced the environments, datasets, benchmarks, andinterfaces to promote research towards developing general virtual agents forthe future.