AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Abstract: Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.

• The paper demonstrates that persistent iteration and empirical feedback loops are key to success on ultra long-horizon research and engineering tasks.
• It introduces AutoLab, a benchmark with 36 curated tasks spanning system optimization, puzzle challenges, ML model development, and CUDA kernel optimization.
• Results reveal claude-opus-4.6 achieves a leading Avg@3 of 0.68, underscoring the value of cost-aware, iterative performance over single-shot optimization.

AutoLab: Evaluating Frontier Models on Ultra Long-Horizon Research and Engineering Tasks

Motivation and Benchmark Design

AutoLab addresses a core deficiency in current LLM evaluation: the inability of most benchmarks to probe long-horizon, closed-loop optimization tasks characteristic of authentic scientific and engineering workflows. Unlike single-shot or short-turn evaluations, ultra long-horizon tasks require agents to execute a persistent cycle of empirical editing, benchmarking, and iteration under explicit wall-clock constraints, mirroring real-world research and systems optimization.

The benchmark comprises 36 highly curated tasks spanning four domains: system optimization, puzzle/content challenge, model development, and CUDA kernel optimization. Each task supplies a correct but deliberately suboptimal baseline, a sealed evaluation environment, calibration-anchored scoring, and a controlled wall-clock budget. Agents operate within containerized sandboxes, modify code, execute empirical evaluations, and must maximize metric improvement within their budget. To ensure defensibility and comparability of results, AutoLab incorporates continuous, hack-resistant scoring and standardized harnesses.

Figure 1: Benchmark results and domain breakdowns across 11 flagship models, with claude-opus-4.6 leading all domains by Avg@3 and Best@3.

Design commitments center on: (1) tasks selected for authentic engineering and scientific difficulty rather than artificial challenge, (2) fine-grained, continuous, and aggregation-friendly scoring (log-stretch and linear anchored to baseline and human reference), and (3) robust anti-hacking via sealed verifiers, correctness gates, SHA-pinning, and adversarial agent auditing.

Figure 2: Task pipeline, specifying input/output interface, sandboxing, empirical feedback, and verification for closed-loop agent operation.

Benchmark Coverage

AutoLab's composition ensures a broad spectrum of algorithmic and practical systems challenges. System optimization tasks demand low-level algorithmic improvement and memory/layout engineering across C, Rust, and Go code. Puzzle tasks emphasize algorithmic creativity and insight. Model development tasks span modern ML pipelines: scalable LLM pretraining, LoRA tuning, reward-based fine-tuning, and online serving. CUDA tasks cover kernel-level performance optimization essential for realistic ML and cryptographic pipelines.

Figure 3: Task distribution illustrating the coverage and diversity of challenge types in AutoLab.

The benchmark includes rigorous multi-round auditing for validity, solvability, integrity, and stability. Artifacts and harnesses are open-sourced to facilitate reproducible research and longitudinal comparison.

Evaluation and Results

Experiments assess 17 SOTA LLMs (both proprietary and open-weight), including claude-opus-4.6, gpt-5.4, gemini-3.1-pro, mimo-v2.5-pro, glm-5, kimi-k2.6, deepseek-v4-pro, among others. Evaluations are conducted under identical hardware and agent harness conditions (terminus-2), reporting average (Avg@3), best (Best@3), and head-to-head Dominance scores.

claude-opus-4.6 consistently outperforms all other models by a significant margin, achieving Avg@3 of 0.68 and Dominance 0.93, as compared to the next-best model (gemini-3.1-pro) at Avg@3 of 0.50.

Analyzing performance by category reveals claude-opus-4.6's superiority is uniform, extending across all domains, with its largest relative lead in CUDA kernel tasks, where many other models score near-zero. Notably, several large proprietary models underperform small open-weight models due to early termination or inefficient budgeting of wall-clock time.

A key empirical finding is that long-horizon task success correlates with the number and persistence of benchmark-edit-iterate steps within the agent, not with the upper quartile quality of its initial solution. Models that persistently exploit empirical feedback loops outperform those that attempt single-shot optimization, even with more advanced base reasoning.

Trajectory and Failure Mode Analyses

Fine-grained trajectory analysis demonstrates that maximal progress requires effective empirical iteration and time-budget awareness.

Figure 4: Best-case optimization trajectories for the flash_attention task reveal that final runtime is fundamentally determined by iterative improvement, with only claude-opus-4.6 exceeding the human reference solution.

Resource utilization analyses show a strong positive correlation between agent steps and final score. claude-opus-4.6 executes substantially more iterative actions (57 median steps) than peers. High scores are generally associated with increased inference/compute expenditure, though some open-weight models like deepseek-v4-flash achieve competitive scores at much lower cost.

Figure 5: Relationship between model performance and resource utilization: agent steps, runtime, and inference cost.

Failure analysis (over 302 zero-scoring rollouts) partitions errors into timeouts/context exhaustion, capability gaps, instruction violations, and extrinsic sandbox failures.

Figure 6: Failure mode distribution by model, revealing the dominance of time management and context control issues among failure classes.

The dominant observed limitation is lack of time-budget calibration: some models terminate after trivial exploration, while others exhaust their entire budget in protracted "thinking loops" without submitting any solution. Instruction-following violations persist even among proprietary models.

Harness and Cost-Performance Trade-Offs

To evaluate the effect of the agent harness, the same base models were scored under mini-swe-agent* (aggressive iteration-prompt), pi-mono, and terminus-2 harnesses. The choice of harness introduces variance on the same order as the difference between base models, and affects not only final score but also total inference cost.

Harnesses designed for persistent iteration substantially raise the attained scores for weaker base models and expose meaningful cost-score Pareto frontiers unseen under fast-exit harnesses.

Figure 7: Per-task sensitivity to harness indicates that harness design can rival model choice in importance, with non-transitive effects by harness and model pair.

Figure 8: Cost vs. outcome frontiers, showing the harness-induced trade-off in compute expenditure for given improvement in performance.

Model Evolution and Stability

Inter-generation analyses within providers reveal model upgrades do not yield universal cross-domain improvement: newer models sometimes regress substantially in specific task families (e.g., Qwen 3.6 Plus on CUDA and system optimization), indicating the necessity for ongoing multi-domain, multi-horizon regression testing.

Figure 9: Model generation delta analysis, showing that improvements are not guaranteed across all evaluation axes.

Stability analysis reveals that high-variance models are over-rewarded by Best@3 and that mean/variance metrics are essential for assessing reliability in ultra-long-horizon tasks.

Theoretical and Practical Implications

AutoLab distinguishes itself by empirically demonstrating that:

• Persistence, iteration, and empirical search, not simply static code generation ability, are the key capabilities underlying success on open-ended, research-style tasks.
• Effective agent operation on these tasks critically depends not only on model internals but on the architecture of agent harnesses and time management policies.
• Model evaluation in the agentic setting must move from final-score “snapshot” metrics to trajectory-sensitive and cost-aware protocols.

These results reinforce the need for future LLMs and agents to incorporate mechanisms for time-budget planning, adaptive persistence, empirical validation, and robust recovery from failed search trajectories.

AutoLab’s open-source release of its tasks, harnesses, and scoring infrastructure enables reproducible research and efficient development of agents aimed at non-trivial scientific and engineering automation.

Conclusion

AutoLab provides a rigorous, multi-domain, ultra long-horizon benchmark that exposes the unique challenges confronting frontier LLM-based agents in real-world research and engineering. The dominant determinants of agent success shift from static competence to persistent, empirically guided search and robust time management. The findings underline the necessity for future work on agent harness architectures, policy iteration scheduling, and interpretability of long-horizon feedback. AutoLab establishes a new standard for diagnostic, robust, and cost-conscious evaluation on tasks that are representative of consequential AI deployment in scientific and engineering practice (2606.05080).