AutoLab: Iterative Code Optimization Benchmark

• AutoLab is a formal benchmark for ultra long-horizon closed-loop optimization, employing iterative improvement cycles on intentionally suboptimal baselines.
• It comprises 36 expert-curated tasks across system optimization, model development, CUDA kernel tuning, and algorithmic challenges with strict wall-clock limits.
• Empirical results emphasize that sustained agent persistence and effective time management are key to outperforming baselines in diverse optimization scenarios.

AutoLab is a formal benchmark and methodology for ultra long-horizon, closed-loop code optimization, designed to evaluate agent and model capabilities in persistent, empirical, iterative improvement scenarios. It encompasses a suite of 36 expert-curated tasks in system optimization, model development, CUDA kernel engineering, and algorithmic challenges. Each task begins from a correct yet intentionally suboptimal baseline and tasks agents with producing empirically superior solutions under strict wall-clock constraints, highlighting the role of sustained, feedback-driven iteration over one-shot generation (Xu et al., 3 Jun 2026).

1. Formal Problem Definition: Ultra Long-Horizon Closed-Loop Optimization

AutoLab frames each task as a closed-loop optimization process. The agent receives a baseline artifact, $x_0$, and, at each iteration $t$, proposes an improved artifact $x_t$ by editing, compiling/executing, measuring performance, and assimilating feedback. The core elements are:

• Baseline artifact: $x_0$ (a working, suboptimal codebase or system)
• Iteration sequence: $\{x_t\}_{t=1}^{N}$ (agent-proposed improvements)
• Performance metric: $m(x)$ (e.g., runtime, perplexity)
• Anchored normalization: For a minimization task, the normalized score is

$$s(x) = \mathrm{clip}\left(\frac{1}{2} \frac{\ln(m_{\mathcal B}/m(x))}{\ln(m_{\mathcal B}/m_{\mathcal R})}, 0, 1\right)$$

where $m_{\mathcal B}$ is baseline and $m_{\mathcal R}$ the human anchor.

• Wall-clock budget: All iterations must complete in allotted time $T$. A hard gating constraint enforces $t$0 unless the baseline is strictly improved. The entire process is terminated upon budget exhaustion, and agents are ranked by $t$1 of their best artifact.

2. Task Suite and Benchmark Composition

AutoLab comprises 36 tasks, classified into four domains:

Domain	Number of Tasks	Score Mapping	Example Metric
System Optimization	15	Log-stretch	Single-thread runtime
Puzzle & Challenge	10	Mixed	Bits/byte, comparator count, etc.
Model Development	7	Anchored linear	Accuracy, perplexity, PSNR
CUDA Kernel Optimization	4	Log-stretch	Kernel runtime (ms)

Each task specifies a baseline, a human reference, and an explicit performance metric. Examples include AES-CTR decryption, FFT computation, agent tool routing (Python), combinatorial sorting networks, and large-scale CUDA kernel tuning.

The normalized scoring functions are tailored per task:

• For unbounded lower-better metrics: anchored log-stretch.
• For bounded (e.g., accuracy): anchored linear scaling.

A strict “must-beat-baseline” requirement enforces non-triviality. Task time budgets span 2–12 hours depending on complexity.

3. Evaluation Protocol, Metrics, and Harness

AutoLab’s evaluation harness executes agent rollouts in a containerized, reproducible environment:

• Execution: Each agent receives the baseline, a description, and a harness orchestrating edit–execute–measure–refine cycles, all through the terminus-2 agent API.
• Rollouts: Three independent rollouts per (agent, task) pair.
• Metrics:
  • $t$2: Mean normalized score over three runs.
  • $t$3: Maximum score over runs.
  • $t$4: Fraction of agent-task pairs in which a model $t$5 outperforms others.
  • Persistence: $t$6 (number of agent steps relative to maximal per-task effort).
  • Time Awareness: Measures how agents utilize their allowed budget, penalizing premature or overly-delayed submission.

4. Empirical Results and Model Comparison

Seventeen models, including 11 proprietary and 6 open-weight systems, were benchmarked on AutoLab. Major quantitative outcomes:

• Claude Opus 4.6 achieved overall $t$7, dominating across all domains ($t$8).
• The strongest open models (Kimi K2.6, MiMo V2.5-Pro, GLM-5) scored in the $t$9–$x_t$0 range, excelling in system but not in CUDA or model development tasks.
• Case study: On flash_attention (C), Opus 4.6 improved the baseline from $x_t$1 ms to $x_t$2 ms in $x_t$3 iterations ($x_t$4 min), surpassing the 100 ms human reference, while other models saturated or stalled.
• Results reveal clear structure–persistence correlation: Models with more steps (e.g., Opus 4.6, mean $x_t$5 steps/task) consistently outscore those with briefer rollouts (e.g., Gemini 3.1 Pro, $x_t$6 steps/task).

Key failure modes across 302 zero-score rollouts included:

• Timeout/context exhaustion (no final submission)
• Capability gap (fails to beat baseline)
• Instruction violation (disallowed APIs)
• Execution/sandbox errors

5. Analysis of Core Insights and Failure Modes

AutoLab provides quantitative evidence that, across domains, sustained agent persistence and effective use of time budget are the critical determinants of success. Specifically:

• Final normalized score correlates more strongly with persistence (number of closed-loop edit–measure cycles) than with initial one-shot solution quality.
• Both “too eager” (low step count, early submit) and “too patient” (exhaust resources without improvement) behaviors yield suboptimal outcomes—time-awareness is thus essential.
• Tooling and harnesses (e.g., prompt engineering, agent wrappers) significantly modulate iteration count and, consequently, task score even for weaker models.

6. Impact, Open Source Artifacts, and Benchmarks for Agent Research

By formalizing and systematically evaluating ultra long-horizon, empirical optimization, AutoLab establishes a new research substrate for assessment and development of agentic systems. All code, evaluation harnesses, sealed verifiers, and task artifacts are open-sourced to maximize reproducibility and community benchmarking:

• GitHub repository: https://github.com/autolabhq/autolab
• Official website: https://autolab.moe The benchmark enables reproducibility auditing (2,544 wall-clock hours, 8.6B evaluation tokens) and provides sealed ground-truth for fair model comparison.

AutoLab fills a critical methodological gap for agentic research and evaluation, explicitly targeting long-horizon, iterative, and feedback-driven progress—the defining characteristics of real-world scientific and engineering domains (Xu et al., 3 Jun 2026).