Abstract
Machine learning, the foundation of modern artificial intelligence, hasdriven innovations that have fundamentally transformed the world. Yet, behindadvancements lies a complex and often tedious process requiring labor andcompute intensive iteration and experimentation. Engineers and scientistsdeveloping machine learning models spend much of their time on trial-and-errortasks instead of conceptualizing innovative solutions or research hypotheses.To address this challenge, we introduce AI-Driven Exploration (AIDE), a machinelearning engineering agent powered by large language models (LLMs). AIDE framesmachine learning engineering as a code optimization problem, and formulatestrial-and-error as a tree search in the space of potential solutions. Bystrategically reusing and refining promising solutions, AIDE effectively tradescomputational resources for enhanced performance, achieving state-of-the-artresults on multiple machine learning engineering benchmarks, including ourKaggle evaluations, OpenAI MLE-Bench and METRs RE-Bench.