OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

Abstract: LLMs have demonstrated remarkable capabilities in reasoning and tool use. However, the fundamental cognitive faculties essential for problem solving, including perception, reasoning, and memory, remain the stable core of intelligence. Unlike memorizing specific patterns, humans succeed in novel environments by applying these intrinsic faculties to adapt and optimize. Yet, whether LLMs possess this essential capacity, namely the ability to continuously refine solutions in response to dynamic environmental feedback, remains underexplored. To address this challenge, we introduce OPT-BENCH, a benchmark for evaluating self-improvement capabilities in large-scale search spaces. By combining 20 machine learning tasks with 10 classic NP-hard problems, OPT-BENCH provides a rigorous setting to assess whether agents can adapt through intrinsic self-reflection rather than rote tool application. We further propose OPT-Agent, a framework that emulates human-like cognitive adaptation. It operates through a general perception, memory, and reasoning loop, iteratively refining solutions based on environmental feedback. Through extensive experiments on 19 LLMs from 7 model families, including reasoning models, general models, and open-source models ranging from 3B to 235B parameters, we demonstrate that stronger models are more effective at leveraging feedback signals for self-improvement. However, this upper-bound adaptability remains fundamentally constrained by the models' base capacity, and even the most advanced LLMs still fall short of human expert performance.

• The paper introduces OPT-BENCH, a 30-environment benchmark and OPT-Agent framework that measure iterative solution improvement across 20 continuous ML tasks and 10 discrete NP-hard problems.
• The evaluation of 19 LLMs shows that agents use numerical feedback effectively—GPT-4o records an 18/2 win count and gpt-4.1 reaches a 2.15 improvement rate—while binary NP feedback often causes random search with memory.
• The findings show that model scale, reasoning ability, and temperature affect performance, but agents still fall short of human experts because they struggle to make targeted structural repairs and sustain long-horizon improvement.

OPT-BENCH is a benchmark and agent framework designed to measure whether LLMs can iteratively improve solutions in response to environmental feedback, rather than merely producing correct answers in a single pass (2605.08904). The work addresses a gap in existing evaluation: most benchmarks, from static knowledge tests to one-shot reasoning suites such as NPHardEval (Fan et al., 2023) and code-generation benchmarks like HumanEval (Chen et al., 2021), assess instantaneous deduction. Even agentic benchmarks such as AgentBench (Liu et al., 2023), WebArena (Zhou et al., 2023), and MLE-Bench (Chan et al., 2024) emphasize task completion or execution success rather than the trajectory of solution refinement over time.

Benchmark design

OPT-BENCH comprises 30 environments spanning two deliberately contrasting feedback landscapes:

• Continuous inductive optimization (20 ML tasks): Kaggle competitions covering regression, classification, and forecasting (e.g., bike-sharing demand, house prices, sentiment analysis), with strictly defined metrics such as RMSE, AUC, and SMAPE. Task descriptions were refined via GPT-4o and verified by human experts; initial "cold start" solutions were generated with the AIDE agent (Jiang et al., 18 Feb 2025) and refined by four PhD-level experts so that the starting point is functional but suboptimal. The human upper bound is the Kaggle gold-medal leaderboard solution.
• Discrete deductive reasoning (10 NP-hard problems): classical combinatorial tasks including TSP, Hamiltonian Cycle, Graph Coloring, Maximum Clique, Knapsack, Set Cover, Minimum Cut, Subset Sum, Maximum Set, and Meeting Scheduling. Validity is enforced by a rule-based validation.py that returns binary feedback plus specific error messages (e.g., "Node visited twice"). The human baseline is a standard expert heuristic (e.g., simulated annealing), not an industrial solver's optimum.

Each sample includes a task description, dataset specification, submission format, initial solution, feedback mechanism, and expert baseline. This design forces agents to switch between inductive reasoning over noisy numerical gradients (ML) and deductive constraint satisfaction over brittle discrete structures (NP).

OPT-Agent

The accompanying OPT-Agent framework implements a perception–memory–reasoning loop inspired by AlphaEvolve (Novikov et al., 16 Jun 2025), deliberately avoiding heavy prompt engineering or external solvers. It operates through three actions:

• Drafting: generating an initial hypothesis — a Python training script for ML, or a directly constructed discrete structure for NP. For NP tasks the model must construct solutions internally rather than offloading logic to a code interpreter.
• Improving: when a valid solution exists, the model retrieves historical context (past solutions, metrics, feedback trends) and proposes atomic refinements.
• Debugging: on failure signals, the model analyzes error logs to diagnose and repair the fault.

Evaluation protocol

Self-optimization is quantified against two bounds: a memory-less random rollout as the lower bound ("blind" guessing) and the human expert as the upper bound. Four metrics are used:

• Win Count: number of tasks where the agent beats the random rollout, measuring whether gains come from genuine feedback use rather than stochastic fluctuation.
• Improvement Rate (IR): mean ratio of post-optimization metric to baseline metric across tasks; IR > 1.0 indicates effective use of environmental feedback.
• Expert Gap (EG): normalized progress between the initial draft and the expert solution, enabling comparison across heterogeneous metrics.
• Buggy Rate: proportion of invalid attempts, particularly diagnostic in NP domains where validity is a hard threshold.

Main findings

Experiments cover 19 LLMs from 7 families (3B–235B parameters), including proprietary, reasoning-enhanced, and open-source models, evaluated at 5, 10, and 20 optimization steps.

Continuous ML domains. Strong models act as effective inductive optimizers. GPT-4o achieves an 18/2 Win Count over the random baseline at 20 steps, confirming history-driven refinement rather than lucky sampling. The paper reports what it terms a "Scaling Law of Self-Improvement": Expert Gap rises sharply with scale (0.45 for Qwen2.5-72B versus 0.20 for Qwen2.5-7B), which the authors attribute to smaller models treating complex error traces as noise due to limited semantic working memory. Reasoning models outperform general counterparts, with gpt-o3-mini reaching the highest EG closure of 0.65, indicating that chain-of-thought reasoning supports deducing causal failure mechanisms rather than guessing parameters. Long horizons reward strong contextual retention: gpt-4.1 attains an IR of 2.15 at 20 steps, while weaker models plateau or regress toward random exploration.

Discrete NP domains. The picture reverses. Win Counts are near-balanced (GPT-4o: 4/6 at 20 steps), revealing a "feedback efficiency paradox": binary error signals such as "Invalid Cycle" lack directional gradients, so self-optimization degenerates into "random search with memory." Reasoning models dominate here — Deepseek-V3.1-Thinking achieves a 0.00 buggy rate and the highest EG (~0.79) — confirming that structured CoT reasoning is essential for maintaining global topological consistency. Smaller open-source models hit a validity bottleneck with buggy rates around 0.80; without crossing this feasibility threshold, historical feedback is useless because no valid baseline exists to optimize from. Notably, even strong reasoning models show limited improvement over steps despite high initial performance, which the authors tentatively attribute to RL-alignment constraints inhibiting long-horizon exploration — an open question rather than an established mechanism.

Temperature ablation. In ML tasks, exploitation dominates: gpt-4o peaks at $T=0$ (13/7 Win Count) and degrades at $T=0.8$ (10/10). In NP tasks, moderate stochasticity helps escape rugged local optima — gpt-4o's buggy rate drops from 0.28 at $T=0$ to 0.18 at $T=0.2$ — but excessive randomness ($T=0.8$) destroys logical coherence and drops Win Count to 4/6.

Draft setting. An appendix ablation shows that generating solutions from scratch can outperform refinement when valid solutions are produced: most tested models achieve higher improvement rates under draft optimization than refine, though open-source models exhibit persistently higher buggy rates, underscoring validity management as the binding constraint.

Discussion: limits of intrinsic self-optimization

The central qualitative finding is a signal-interpretation divergence. LLMs map semantic or numerical feedback to coherent incremental updates in ML, but fail to perform incremental structural editing in NP domains: in the Hamiltonian cycle task, where a human would locally repair a disconnected edge, models frequently discard accumulated history and regenerate entire solutions. Across both domains, a persistent human gap remains — agents improve over their drafts but rarely reach expert-derived optima, suggesting current self-optimization behaves as local search bounded by the model's inherent reasoning depth rather than global planning.

Limitations and open questions

The authors concede several constraints explicitly. Resource limitations excluded newer frontier models (Gemini 3 series, GPT-5.2, Claude 4.5) from evaluation. The benchmark contains only 30 environments, limiting statistical robustness per task family. The explanation for reasoning models' flat long-horizon improvement in NP tasks is offered as a hypothesis, not verified. More broadly, the finding that discrete error signals resist translation into structural repairs leaves open how feedback representations could be reformatted to support deductive refinement without external solvers.

Conclusion

OPT-BENCH provides a controlled contrast between continuous and discrete optimization landscapes for evaluating iterative self-improvement in LLM agents. Its results establish that feedback-driven self-optimization scales with base model capability and benefits substantially from chain-of-thought reasoning, but is fundamentally asymmetric: numerical gradients support reliable incremental refinement, while brittle combinatorial constraints do not. The benchmark shifts evaluation from "can the agent run the code?" to "can the agent evolve the solution?", and its persistent expert gap quantifies how far current intrinsic self-optimization remains from principled heuristic reasoning.