---
title: 'AutoLab: Iterative Code Optimization Benchmark'
url: https://www.emergentmind.com/topics/autolab
type: topic
---

# AutoLab: Iterative Code Optimization Benchmark

AutoLab is a formal benchmark and methodology for ultra long-horizon, closed-loop code optimization, designed to evaluate agent and model capabilities in persistent, empirical, iterative improvement scenarios. It encompasses a suite of 36 expert-curated tasks in system optimization, model development, CUDA kernel engineering, and algorithmic challenges. Each task begins from a correct yet intentionally suboptimal baseline and tasks agents with producing empirically superior solutions under strict wall-clock constraints, highlighting the role of sustained, feedback-driven iteration over one-shot generation [2606.05080].

## 1. Formal Problem Definition: Ultra Long-Horizon Closed-Loop Optimization

AutoLab frames each task as a closed-loop optimization process. The agent receives a baseline artifact, $x_0$, and, at each iteration $t$, proposes an improved artifact $x_t$ by editing, compiling/executing, measuring performance, and assimilating feedback. The core elements are:
- **Baseline artifact:** $x_0$ (a working, suboptimal codebase or system)
- **Iteration sequence:** $\{x_t\}_{t=1}^{N}$ (agent-proposed improvements)
- **Performance metric:** $m(x)$ (e.g., runtime, perplexity)
- **Anchored normalization:** For a minimization task, the normalized score is
  \[
  s(x) = \mathrm{clip}\left(\frac{1}{2} \frac{\ln(m_{\mathcal B}/m(x))}{\ln(m_{\mathcal B}/m_{\mathcal R})}, 0, 1\right)
  \]
  where $m_{\mathcal B}$ is baseline and $m_{\mathcal R}$ the human anchor.
- **Wall-clock budget:** All iterations must complete in allotted time $T$.
A hard gating constraint enforces $s(x) = 0$ unless the baseline is strictly improved. The entire process is terminated upon budget exhaustion, and agents are ranked by $s(x)$ of their best artifact.

## 2. Task Suite and Benchmark Composition

AutoLab comprises 36 tasks, classified into four domains:

| Domain                | Number of Tasks | Score Mapping    | Example Metric                      |
|-----------------------|:---------------:|-----------------|--------------------------------------|
| System Optimization   | 15              | Log-stretch      | Single-thread runtime                |
| Puzzle & Challenge    | 10              | Mixed            | Bits/byte, comparator count, etc.    |
| Model Development     | 7               | Anchored linear  | Accuracy, perplexity, PSNR           |
| CUDA Kernel Optimization | 4            | Log-stretch      | Kernel runtime (ms)                  |

Each task specifies a baseline, a human reference, and an explicit performance metric. Examples include AES-CTR decryption, FFT computation, agent tool routing (Python), combinatorial sorting networks, and large-scale CUDA kernel tuning.

The normalized scoring functions are tailored per task:
- For unbounded lower-better metrics: anchored log-stretch.
- For bounded (e.g., accuracy): anchored linear scaling.

A strict “must-beat-baseline” requirement enforces non-triviality. Task time budgets span 2–12 hours depending on complexity.

## 3. Evaluation Protocol, Metrics, and Harness

AutoLab’s evaluation harness executes agent rollouts in a containerized, reproducible environment:
- **Execution:** Each agent receives the baseline, a description, and a harness orchestrating edit–execute–measure–refine cycles, all through the terminus-2 agent API.
- **Rollouts:** Three independent rollouts per (agent, task) pair.
- **Metrics:**
  - $\mathrm{Avg@3}$: Mean normalized score over three runs.
  - $\mathrm{Best@3}$: Maximum score over runs.
  - $\mathrm{Dom}(m)$: Fraction of agent-task pairs in which a model $m$ outperforms others.
  - **Persistence:** $\mathrm{Pers}(m) = \frac{1}{|\mathcal T|}\sum_{t}\frac{N_{m,t}}{\max_{m'}N_{m',t}}$ (number of agent steps relative to maximal per-task effort).
  - **Time Awareness:** Measures how agents utilize their allowed budget, penalizing premature or overly-delayed submission.

## 4. Empirical Results and Model Comparison

Seventeen models, including 11 proprietary and 6 open-weight systems, were benchmarked on AutoLab. Major quantitative outcomes:
- **Claude Opus 4.6** achieved overall $\mathrm{Avg@3}=0.68$, dominating across all domains ($\mathrm{Dom}=0.93$).
- The strongest open models (Kimi K2.6, MiMo V2.5-Pro, GLM-5) scored in the $0.43$–$0.46$ range, excelling in system but not in CUDA or model development tasks.
- Case study: On flash_attention (C), Opus 4.6 improved the baseline from $750$ ms to $18$ ms in $~44$ iterations ($\sim40$ min), surpassing the 100 ms human reference, while other models saturated or stalled.
- Results reveal clear structure–persistence correlation: Models with more steps (e.g., Opus 4.6, mean $\sim57$ steps/task) consistently outscore those with briefer rollouts (e.g., Gemini 3.1 Pro, $\sim12$ steps/task).

Key failure modes across 302 zero-score rollouts included:
- Timeout/context exhaustion (no final submission)
- Capability gap (fails to beat baseline)
- Instruction violation (disallowed APIs)
- Execution/sandbox errors

## 5. Analysis of Core Insights and Failure Modes

AutoLab provides quantitative evidence that, across domains, sustained agent persistence and effective use of time budget are the critical determinants of success. Specifically:
- Final normalized score correlates more strongly with persistence (number of closed-loop edit–measure cycles) than with initial one-shot solution quality.
- Both “too eager” (low step count, early submit) and “too patient” (exhaust resources without improvement) behaviors yield suboptimal outcomes—time-awareness is thus essential.
- Tooling and harnesses (e.g., prompt engineering, agent wrappers) significantly modulate iteration count and, consequently, task score even for weaker models.

## 6. Impact, Open Source Artifacts, and Benchmarks for Agent Research

By formalizing and systematically evaluating ultra long-horizon, empirical optimization, AutoLab establishes a new research substrate for assessment and development of agentic systems. All code, evaluation harnesses, sealed verifiers, and task artifacts are open-sourced to maximize reproducibility and community benchmarking:
- GitHub repository: https://github.com/autolabhq/autolab
- Official website: https://autolab.moe
The benchmark enables reproducibility auditing (2,544 wall-clock hours, 8.6B evaluation tokens) and provides sealed ground-truth for fair model comparison.

AutoLab fills a critical methodological gap for agentic research and evaluation, explicitly targeting long-horizon, iterative, and feedback-driven progress—the defining characteristics of real-world scientific and engineering domains [2606.05080].

Source: https://www.emergentmind.com/topics/autolab