AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

Key points

• Problem: Language model (LM) agents called AI co-scientists automate various research steps from hypothesis generation to experiment execution, but judging the scientific contribution of these agents is hard. The authors treat the planning step of ablation experiments, which measure the contribution of each method component, as the key capability. The earlier AbGen builds plans from manually processed paper content and relies mainly on human expert ranking, which leaves fully automated ablation planning from raw papers unexplored.
• Input and output: This is a benchmark paper. AuthorAblation gives the title, abstract, and LaTeX source of a paper cut after the method section and asks for up to 5 ablations in JSON Lines (JSONL); it has 83 papers and 230 gold ablations. ReviewerAblation gives the full submission and asks for up to 2 missing ablations, using the official reviews of 350 ICLR submissions as gold labels.
• Method: An LM judge matches the ablations in a generated plan against the gold ablations and computes precision@k, recall@k, and F1@k. The judge has two forms, LMJUDGE with a single chain-of-thought (CoT) prompt and AGENTJUDGE taking multiple steps in SWE-agent, and the baseline planners use the same two forms, LM-PLANNER and AGENT-PLANNER. Judge bias is reduced by a three-model majority vote and shuffled ablation order, and in AuthorAblation the tags that mark gold and plan are removed and the two sides are randomly assigned as Side A and Side B.
• Implementation: AuthorAblation was drawn from 203 papers from three earlier benchmarks plus CVPR and ICCV best papers from 2020 to 2024, and ReviewerAblation from the 5,960 papers left after filtering reviews of about 22,800 ICLR submissions with an 'ablat' keyword search and a GPT-4o CoT filter. The planner experiments used seven LMs: GPT-5.4, o3-mini, Gemini 2.5 Flash, Llama 3.1 405B, GPT-OSS-120B, MiniMax M2.5, and Qwen 3.5 122B. The official GitHub repository holds the ablation-bench command-line interface (CLI) with plan, eval, and eval-judge commands, and its README states that the four datasets are on Hugging Face.
• Results: On the AblationBench F1@k averaged over the two tasks, GPT-5.4 scored highest within each scaffold at 0.35 for both LM-PLANNER and AGENT-PLANNER, with recall@k of 0.45 and 0.44 respectively. On papers that 10 machine learning PhD participants each chose from 15 randomly selected AuthorAblation test papers, the participants reached F1@5 of 0.66, while GPT-5.4 scored 0.43 on the same papers. On JudgeEval, the judge evaluation set, the LMJUDGE majority vote reached F1 0.79 and kappa 0.62 against human labels, above the AGENTJUDGE majority vote at F1 0.76 and kappa 0.57.
• Limits: The authors state that the evaluation relies on LM judges which, despite strong classification performance, still leave room for improvement. Meaningful and feasible ablations outside the gold set count as unmatched proposals, and evaluating such new proposals is left for future work. The benchmark covers the ablation planning step; running the plans as code and collecting results is outside the focus of this paper.

Paper links

External research summaries. These are not HDATF publications or measured product results.