Abstract
We introduce AbGen, the first benchmark designed to evaluate the capabilitiesof LLMs in designing ablation studies for scientific research. AbGen consistsof 1,500 expert-annotated examples derived from 807 NLP papers. In thisbenchmark, LLMs are tasked with generating detailed ablation study designs fora specified module or process based on the given research context. Ourevaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights asignificant performance gap between these models and human experts in terms ofthe importance, faithfulness, and soundness of the ablation study designs.Moreover, we demonstrate that current automated evaluation methods are notreliable for our task, as they show a significant discrepancy when compared tohuman assessment. To better investigate this, we develop AbGen-Eval, ameta-evaluation benchmark designed to assess the reliability of commonly usedautomated evaluation systems in measuring LLM performance on our task. Weinvestigate various LLM-as-Judge systems on AbGen-Eval, providing insights forfuture research on developing more effective and reliable LLM-based evaluationsystems for complex scientific tasks.