AblationBench: AI Ablation Planning Benchmark

• AblationBench is a benchmark suite for evaluating ablation planning in empirical AI, converting method narratives into structured control experiments.
• It defines two tasks—AuthorAblation and ReviewerAblation—with a standardized JSONL schema to generate and rank proposed ablation experiments.
• Empirical results show that state-of-the-art LM systems identify only about 29% of method ablations, highlighting significant challenges in experimental reasoning.

Searching arXiv for the benchmark and closely related work to ground the article in current papers. AblationBench is a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It targets the upstream reasoning involved in ablation studies: given a method, deciding which components to ablate, how to ablate them, and what metrics to report. The benchmark is defined through two tasks, AuthorAblation and ReviewerAblation, paired with LM-based automatic judges and structured JSONL outputs. In reported experiments, frontier LM systems remain substantially limited on these tasks, with the best-performing system identifying only about $29\%$ of the original ablations on average across tasks, including approximately $34\%$ on AuthorAblation and $26\%$ on ReviewerAblation (Abramovich et al., 9 Jul 2025).

1. Scope, problem definition, and naming

In empirical AI research, ablation studies isolate the contribution of individual components of a method—modules, design choices, losses, and hyperparameters—by removing, replacing, or altering them and measuring performance impact. Ablation planning is the reasoning stage that precedes those experiments. It requires identifying method components at an appropriate granularity, proposing relevant and feasible controls, mapping informal paper text into structured experimental proposals, and judging whether a control is meaningful rather than trivial (Abramovich et al., 9 Jul 2025).

The benchmark addresses two research workflows. For authors, ablation planning turns a method narrative into a credible set of control experiments; for reviewers, identifying missing ablations is central to assessing whether empirical validation is sufficient and whether claimed contributions are robust. AblationBench formalizes both perspectives into benchmark tasks, rather than treating ablation design as an unstructured prompting exercise.

A common source of confusion is the word ablation itself. In AblationBench, the term refers to empirical AI control experiments, not material erosion or thermo-chemical surface recession. A separate work on ICP wind tunnels uses the name in benchmark design guidance for thermo-chemical ablation modeling, which reflects a distinct technical sense of the term (Kumar et al., 17 Feb 2026). Within AI evaluation, however, AblationBench denotes the benchmark introduced by Talor Abramovich and Gal Chechik for automated planning of ablation experiments (Abramovich et al., 9 Jul 2025).

2. Task formulation and output schema

AblationBench is organized around two precisely defined tasks.

Task	Input	Output
AuthorAblation	paper title, abstract, and the full source up to and including the method section	up to $k$ ablations in JSONL
ReviewerAblation	paper title, abstract, and full preprint in Markdown	up to $k$ missing ablations in JSONL

For AuthorAblation, the paper is truncated at the method section, with no experiments or results. The objective is to generate a ranked plan of ablation experiments that test the method’s major components. For ReviewerAblation, the objective is to propose missing ablations that should have been conducted but were not, in alignment with typical reviewer suggestions (Abramovich et al., 9 Jul 2025).

The JSONL schema is identical across tasks. Each entry contains "name", "ablated_part", "action" in $\{\text{REMOVE}, \text{REPLACE}, \text{ADD}\}$, an optional "replacement" list for REPLACE or ADD, and "metrics" to report. Entries are ranked by importance. The benchmark fixes $k = 5$ for AuthorAblation and $k = 2$ for ReviewerAblation.

This formulation makes ablation planning machine-evaluable without collapsing it to free-form text generation. The benchmark therefore emphasizes structured correspondence between generated proposals and gold labels, especially component identity, action type, and replacement content when applicable.

3. Dataset construction and benchmark composition

AuthorAblation is a human-curated dataset of 83 papers across 14 conferences, with 230 human-annotated gold ablations and a split of 21 development instances and 62 test instances (Abramovich et al., 9 Jul 2025). Its sources were aggregated from CSR-Bench, SUPER-Expert, PaperBench, and 38 CVPR/ICCV best paper awardees from 2020–2024, then filtered for presence of ablations and public code. The construction procedure filtered papers using the keyword "ablat" followed by manual validation, created truncated papers containing sections up to and including the method section, and manually extracted the ground-truth ablation plan from the full paper into structured JSON.

The dataset statistics are intended to reflect the combinatorial structure of realistic ablation design. Mean and median ground-truth ablations per paper are 3.7 and 3. The gold set contains $59\%$ modification ablations (REPLACE or ADD) and $41\%$ removals, and modification ablations often have multiple valid alternatives, with 3.3 alternatives on average. Mean and median truncated-paper TeX file counts are 4.2 and 2, and mean and median figure counts are 6.8 and 4.

ReviewerAblation contains 350 ICLR submissions, split into 50 development and 300 test instances, sampled from a larger pool of 5,960 candidate papers from ICLR 2023–2025 whose reviews mention ablations. The source pool was obtained from approximately 89,100 reviews across approximately 22,800 submissions. Construction filtered reviews containing the substring "ablat", used a CoT-prompted LM filter to identify reviews with concrete suggestions for new ablations, and converted retrieved PDFs to Markdown using the marker tool. Average reviewer-suggested ablations per paper are approximately 2, and more than half of the papers have only one suggestion. The test-set acceptance rate is approximately $34\%$0, matching the overall ICLR rate.

Quality control is mediated through JudgeEval datasets rather than reported inter-annotator agreement. AuthorEval contains 63 plans with correct match labels against ground-truth ablations, and ReviewerEval contains 60 plans with correct match labels against reviewer suggestions. No explicit inter-annotator agreement metric is reported; validation is instead based on comparing LM judges against these human-labeled sets.

4. Evaluation framework and LM-based judges

AblationBench couples the planning tasks to automatic judges that score semantic matches between generated proposals and gold labels (Abramovich et al., 9 Jul 2025). For AuthorAblation, a generated ablation matches a gold ablation if the ablated_part matches, the action matches, and, for REPLACE or ADD, the replacement content matches; if the ground truth contains multiple valid replacements, any one correct replacement suffices. For ReviewerAblation, a generated proposal matches if it captures the same underlying idea and aligns on the ablated component; action and replacement details need only match when the review explicitly states them.

Planner evaluation uses macro-averaged $34\%$1, $34\%$2, and $34\%$3 across instances:

$34\%$4

$34\%$5

$34\%$6

The overall AblationBench score is the average across the two tasks.

Judge evaluation uses macro-averaged Precision, Recall, and F1 on AuthorEval and ReviewerEval, along with cost per instance. Two judge designs are defined. LMJudge uses a single CoT prompt and outputs a discussion plus predictions in strict JSONL format. AgentJudge is implemented with SWE-agent in a ReAct-style setting, with file-creation and editing tools, command execution, explicit environment constraints, and mandatory initialization of /repo/final_score.jsonl via a create_final_score tool.

On JudgeEval, LMJudge outperforms AgentJudge in accuracy and cost and is therefore selected for scoring the benchmark. Representative macro-averaged results are reported as follows: for LMJudge, GPT-4o obtains Precision 0.74, Recall 0.79, F1 0.73 at cost $34\%$70.02; <a href="https://www.emergentmind.com/topics/claude-3-5-sonnet" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Claude 3.5 Sonnet</a> obtains 0.76, 0.79, 0.75 at $34\%$80.13; o3-mini obtains 0.56, 0.75, 0.61 at $34\%$90.06. Final benchmark scoring uses LMJudge with Claude 3.5 Sonnet.

5. Experimental setup and empirical findings

The benchmark evaluates two planner families: LM-Planner, which uses a single CoT prompt, and Agent-Planner, which uses SWE-agent in a controlled container environment with simple tools (Abramovich et al., 9 Jul 2025). Planner experiments include GPT-4o, Claude 3.5 Sonnet, Llama 3.1 405B Instruct, with LM-Planner additionally including o3-mini and Gemini 2.5 Flash. Cost per instance is tracked via OpenRouter provider pricing. Statistical analysis includes paired $26\%$0-tests for LM-versus-agent comparisons and Spearman correlation for trajectory length.</p> <p>Overall macro-averaged planner results show that LM-Planner is consistently stronger and cheaper than Agent-Planner. Reported overall results are:</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Planner</th> <th>Model</th> <th>Precision@k / Recall@k / F1@k</th> </tr> </thead><tbody><tr> <td>LM-Planner</td> <td>GPT-4o</td> <td>0.22 / 0.27 / 0.22</td> </tr> <tr> <td>LM-Planner</td> <td>Claude 3.5 Sonnet</td> <td>0.20 / 0.26 / 0.21</td> </tr> <tr> <td>LM-Planner</td> <td>Llama 3.1 405B</td> <td>0.18 / 0.24 / 0.19</td> </tr> <tr> <td>LM-Planner</td> <td>o3-mini</td> <td>0.21 / 0.28 / 0.22</td> </tr> <tr> <td>LM-Planner</td> <td><a href="https://www.emergentmind.com/topics/gemini-2-5" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Gemini 2.5</a> Flash</td> <td>0.23 / 0.29 / 0.24</td> </tr> <tr> <td>Agent-Planner</td> <td>GPT-4o</td> <td>0.18 / 0.20 / 0.18</td> </tr> <tr> <td>Agent-Planner</td> <td>Claude 3.5 Sonnet</td> <td>0.19 / 0.23 / 0.19</td> </tr> <tr> <td>Agent-Planner</td> <td>Llama 3.1 405B</td> <td>0.17 / 0.18 / 0.17</td> </tr> </tbody></table></div> <p><a href="https://www.emergentmind.com/topics/gemini-gemini-2-5-pro" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Gemini</a> 2.5 Flash achieves the best overall Recall@k and F1@k at the lowest reported cost, with 0.29 recall, 0.24 F1, and $0.01 per instance. The headline finding is that even the best-performing LM system identifies only about $26\%$1 of original ablations on average across tasks.

Per-task results sharpen the picture. On AuthorAblation, the top recalls are 0.34 for o3-mini and 0.32 for Gemini Flash at $26\%$2. Agent-Planner is substantially weaker; for example, GPT-4o as an agent achieves Recall@5 of 0.19 and F1@5 of 0.17. The difference is statistically significant: LM-Planner with Claude 3.5 Sonnet achieves F1 six percentage points higher than the best agent model, with $26\%$3 in a paired $26\%$4-test on paper-level F1. On ReviewerAblation, LM-Planner reaches Recall@2 of 0.26 with GPT-4o, 0.25 with Gemini Flash, and 0.25 with Llama 3.1, while Agent-Planner attains similar recall at approximately three times higher cost.

Several additional empirical patterns are reported. Removal-type ablations are easier than modification-type ablations across all models, and for GPT-4o the removal-versus-modification recall difference is significant with $26\%$5. Increasing $26\%$6 on AuthorAblation slightly lowers precision and raises recall; for o3-mini, the best trade-off occurs at $26\%$7. NLP conference papers yield lower scores than General ML and CV categories. In ReviewerAblation, there is no clear correlation between recall and year, acceptance status, number of reviews, or review length.

6. Failure modes, practical use, and significance

The benchmark identifies several recurring failure modes (Abramovich et al., 9 Jul 2025). Models often miss core components of a contribution and instead propose ablations over fine-grained implementation details. They also exhibit granularity mismatch, producing proposals that are either too narrow to correspond to the gold ablations or so broad that they do not map cleanly to the intended control. For REPLACE or ADD, weak control choices are common: proposed alternatives are not realistic or not community-standard, so they fail to match the gold labels. In ReviewerAblation, models may miss the exact alignment of a reviewer’s request when the review specifies a particular action or replacement.

The paper’s representative GPT-GNN example makes this failure mode concrete. Ground-truth ablations include removing the two main components, “attribute generation” and “edge generation,” swapping GNN architectures such as GCN, GAT, RGCN, and HAN, and varying the data-label fraction. The LM-Planner using o3-mini instead proposed finer-grained changes such as removing dependency-aware factorization, unifying node types, removing adaptive negative queue, varying masked versus observed edge ratios, and swapping subgraph sampling; the judge found that none of these matched the primary ground-truth ablations. This suggests that current systems are often better at enumerating implementation perturbations than at recovering the method-level controls that matter scientifically.

From a usage perspective, the benchmark prescribes a concrete evaluation workflow. New systems choose AuthorAblation, ReviewerAblation, or both; consume the specified inputs; and output ranked JSONL files with up to five or two entries, respectively. For LM-based judging, outputs must conform exactly to the expected schema. For agent-based judging, the environment uses create_final_score and submit, while planner agents write /repo/ablations_plan.jsonl and submit. Evaluation with LMJudge, recommended with Claude 3.5 Sonnet, returns macro-averaged Precision@k, Recall@k, and F1@k; if both tasks are run, the overall score is their average.

The broader significance of AblationBench lies in what it measures rather than in absolute scores. It is presented as the first benchmark explicitly centered on ablation planning from both authorship and reviewing perspectives, complementing execution-oriented benchmarks such as SUPER-Expert, CSR-Bench, PaperBench, MLAgentBench, and MLGym, as well as literature and evidence tools such as CiteME, LitSearch, and PaperQA2. Its results indicate that current frontier LMs still have substantial room for improvement in experimental reasoning, especially for realistic modification ablations and reviewer-aligned controls. At the same time, the finding that single-shot CoT prompting outperforms current agent scaffolding suggests that these tasks are short-context reasoning problems for which extra tool use adds overhead without supplying the domain-specific representations needed for better ablation planning.

7. Limitations and future directions

AblationBench has several explicitly stated limitations (Abramovich et al., 9 Jul 2025). AuthorAblation focuses on ML, NLP, and CV conferences with available code and explicit ablation sections, so methods outside standard empirical AI publication patterns are underrepresented. ReviewerAblation is centered on ICLR 2023–2025; extension to other venues and years is described as feasible through the automatic pipeline but has not yet been carried out.

The evaluation protocol is also intentionally narrow. Reliance on LM judges introduces the possibility of misclassification, even though the judges are calibrated against human-labeled JudgeEval sets. Ground-truth incompleteness remains a structural issue: a system may propose a meaningful and feasible ablation that is not present in the gold labels, but recall@k does not reward such creativity. Likewise, the current matching criteria require tight alignment on components and actions, so semantically close proposals may be undervalued when component naming, replacement specification, or granularity differs.

The paper identifies several future directions. These include improved judges with specialized taxonomies of method components, synonym and semantic handling, and retrieval of canonical replacements; extension to integrated research pipelines that link planning to code-generation and execution benchmarks; and expansion to broader domains with ablation-type taxonomies such as architecture, training regime, data, and evaluation. A plausible implication is that AblationBench will be most useful not as a terminal scorecard for general scientific agency, but as a controlled test of whether systems can convert method descriptions and review text into experimentally meaningful control plans.