Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

Abstract: Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and longer-horizon trajectories make credit assignment in RL substantially harder. This paper argues that evolution strategies (ES) can be a better choice for fine-tuning long-horizon LLM agents. Compared with agentic RL, ES offers three key advantages: 1) Model Scalability: ES enables full-parameter optimization with only minimal, inference-level GPU memory, making it possible to fine-tune large LLMs. 2) Flexibility: its lightweight, black-box feedback interface makes ES fine-tuning easy to compose with prompt-space evolution (e.g., skill optimization & test-time compute); and 3) Long-Horizon Scalability: ES performs trajectory-level parameter attribution without decomposing rewards across horizons, yielding better scalability than Agentic RL as the horizon length grows. Based on this insight, we propose Agentic ESOpt, a full-parameter agentic fine-tuning framework tailored to flexible parameter--context co-evolution. At each step, Agentic ESOpt samples perturbations around the current LLM parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. To improve the exploration--adaptation trade-off, Agentic ESOpt further introduces a cosine decay schedule of the perturbation scale $σ$. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performs online prompt--parameter co-evolution, improving its matched baseline in 28 of 36 settings.

• The paper introduces Agentic ESOpt, a forward-only evolution-strategy method that fine-tunes long-horizon agents without the activation, optimizer-state, and reference-model memory required by agentic reinforcement learning.
• Controlled Sudoku experiments show a horizon-dependent crossover: at 15 masked cells, Agentic ESOpt reaches 53.13% success versus 40.63% for GRPO and 0% for PPO, while using 85.7% less GPU memory than GRPO.
• The method improves tool-use benchmarks, automatic heuristic design, and large-model WebArena performance, but its benefits depend on rollout cost, population size, hyperparameter tuning, and unresolved continual-learning behavior.

Motivation and central claim

The paper argues that evolution strategies (ES), rather than reinforcement learning, are the better-matched optimization mechanism for fine-tuning long-horizon LLM agents. The authors identify two structural weaknesses of agentic RL in this regime: full-parameter training requires storing activations, optimizer states, and reference models for backpropagation through trajectories, which becomes impractical at larger model scales; and sparse terminal rewards must be attributed across an expanding sequence of turns, degrading credit assignment as horizons grow. Against these, ES offers three properties: model scalability (forward-only updates requiring only inference-level GPU memory), flexibility (a black-box scalar-reward interface that composes with prompt-space and test-time methods), and long-horizon scalability (trajectory-level parameter attribution without per-turn score accumulation). The paper is careful to note that prior ES work on single-turn reasoning found ES to be a cheaper but slightly weaker alternative to RL; the claim here is that the ordering reverses specifically in long-horizon agentic settings.

Method

Agentic ESOpt optimizes a Gaussian-smoothed expected return $$J_\sigma(\theta;c)=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}[J(\theta+\sigma\epsilon;c)]$$ via the standard ES pseudo-gradient $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$, estimated with $G$ perturbations scored by environment rollouts. Rewards are z-score normalized within the population; the implemented update omits the explicit $1/\sigma$ factor, with learning rate $\alpha$ serving as effective scale. Following prior work, only noise seeds are stored and perturbations are reconstructed via in-place addition/subtraction, so training memory equals inference memory.

Two design elements distinguish the method. First, a cosine decay schedule on the perturbation radius $\sigma_t$, motivated by a lemma showing that Gaussian smoothing introduces bias of order $\frac{\sigma^2}{2}\mathrm{Tr}(\nabla_\theta^2 J)$, which acts as flatness regularization. Train-time runs retain a nonzero terminal radius $\sigma_T$; test-time runs decay $\sigma_T$ to zero to remove smoothing bias at convergence. Second, a prompt–parameter co-evolution interface: because the update consumes only trajectories and scalar scores already collected by skill optimizers (Trace2Skill) or evolutionary heuristic search (EoH), parameter adaptation can be interleaved with external-context optimization without modifying those scaffolds.

Long-horizon scaling argument and controlled Sudoku study

The theoretical core is a variance comparison under weak-correlation assumptions inherited from the OpenAI ES analysis: a trajectory-level policy-gradient estimator has variance growing approximately linearly in the realized horizon $H$, since it sums $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$0 action-score terms weighted by return variance, whereas the ES estimator's score factor $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$1 contains no sum over turns. The authors explicitly scope this claim: it isolates horizon-dependent score structure only; ES can still degrade with $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$2 through sparser returns, and total variance depends on dimension, $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$3, $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$4, and local geometry. They also concede that increasing the minimum successful horizon $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$5 changes task difficulty beyond pure delay, so Sudoku tests a practical prediction rather than identifying a pure delay effect.

The controlled multi-turn Sudoku environment masks 5, 10, or 15 cells, fixing $$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$6 with binary terminal reward. Results on Qwen3.5-4B exhibit the predicted crossover:

Method	GPU memory	$$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$7	$$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$8	$$\nabla_\theta J_\sigma = \frac{1}{\sigma}\mathbb{E}[J(\theta+\sigma\epsilon)\epsilon]$$9
Qwen3.5-4B base	8.41 GB	63.54	31.25	10.42
+ Agentic PPO	89.40 GB	90.63	56.25	0.00
+ Agentic GRPO (stronger config)	58.88 GB	85.42	67.71	40.63
+ Agentic ESOpt ($G$0)	8.41 GB	89.58	62.50	53.13

PPO collapses entirely at $G$1 under sparse terminal reward—its critic never learns a reliable value signal—and GRPO variants stall near the 45-turn budget while Agentic ESOpt stays close to the 15-turn minimum (ending at 15.41 turns). The ordering reversal across horizons supports a regime-dependent advantage rather than a uniformly stronger optimizer. Ablations confirm both components matter: vanilla fixed-$G$2 ES reaches 42.71% at $G$3, and setting $G$4 drops to 28.13% due to overfitting. On efficiency, Agentic ESOpt requires 85.7% less GPU memory than GRPO; although it evaluates four times more directions ($G$5 vs. 8 rollouts), measured wall-clock time on four H100s is lower (9.4 h vs. 19.0 h at $G$6), because saved backward-pass and reference-model compute offset the larger population.

Train-time fine-tuning results

On ReAct-style tool use, Agentic ESOpt improves Qwen3.5-4B by an average of 13.7 points over the base model and 8.3 points over matched Agentic GRPO across DAPO Mean@4 (+13.8), AIME 2026 Mean@4 (+15.0), and DocVQA accuracy (+12.3). Notably, gains extend to Pass@K metrics up to $G$7: ESOpt beats GRPO on every reported Pass@4 metric and exceeds even the base model's coverage on AIME and DocVQA, indicating the improvement does not come from collapsing repeated-sampling diversity—a contrast with common RL pathologies. On model-side FLOPs accounting, ESOpt uses roughly half of GRPO's cost on these tasks (one forward pass per trajectory versus approximately four forward-pass equivalents).

On WebArena-Lite, the memory advantage becomes a feasibility result: full-parameter agentic RL of Qwen3.5-27B is impractical on four H100 80GB GPUs, whereas Agentic ESOpt performs full-parameter adaptation within inference-level memory, raising the No Skill baseline from 29.47% to 36.16% (+6.69 points)—above GPT-5.4's 34.14% reference—and improving Trace2Skill from 33.94% to 36.36%. This experiment is framed as feasibility-oriented rather than a controlled ES-versus-RL comparison, since no RL baseline was run at 27B scale.

Test-time automatic heuristic design

Inserted into unmodified Sample and EoH scaffolds with LLaMA-3.1-8B-Instruct under matched evaluation budgets, Agentic ESOpt improves 21 of 24 constructive comparisons (with one tie and two regressions) and 28 of 36 comparisons overall including ACO-style settings. Repeated-run analysis on TSP and KP yields one-sided $G$8-test $G$9-values of 0.0258 and 0.0100, supporting consistency beyond single seeds. Component ablations show that noise-only perturbation without the reward-weighted update, or without the cosine schedule, recovers most but not all of the gain, and retuning EoH sampling temperature does not reproduce the improvement. Runtime overhead is modest: 9.7–18.0% added wall-clock time, confirming updates occur nearly on-the-fly.

Population sensitivity

A preliminary study on 15-turn Sudoku suggests population requirements shrink with backbone strength: doubling $1/\sigma$0 from 8 to 16 raises final test success by +677.0% relative for the 4B model but 0.0% for the 9B model. The proposed interpretation—that useful perturbation directions are denser around stronger pretrained weights—is consistent with related findings but remains an interpretation; the paper explicitly defers establishing a universal population-scaling law to future work.

Limitations and open questions

The paper concedes several limitations directly. Agentic ESOpt introduces hyperparameters ($1/\sigma$1, $1/\sigma$2, $1/\sigma$3, $1/\sigma$4) whose optima may be task- and model-dependent, though reported configurations are fairly consistent across five settings ($1/\sigma$5, $1/\sigma$6). More substantively, the method trades backpropagation for more environment evaluations, a trade-off that inverts when rollout cost dominates—for expensive-to-evaluate environments, the advantage may disappear. Continual-learning behavior also remains unclear: unlike GRPO's sparse updates, dense ES updates induce random-walk components irrelevant to the objective; the authors partially mitigate this concern empirically, showing 96.26% of final WebArena parameter updates fall within the perturbation scale, but do not resolve it. Finally, the 27B WebArena result lacks an RL control at matched scale, and the horizon-variance theory rests on assumptions (weak action–return correlation, uncorrelated per-step scores) that hold only approximately in practice.

Conclusion

This paper positions evolution strategies not as a cheaper substitute for agentic RL but as structurally better suited to long-horizon, sparse-feedback agent fine-tuning. The evidence is strongest where the paper controls the comparison: the horizon-dependent crossover on Sudoku, consistent Pass@K-preserving gains on Math and DocVQA, and significant AHD improvements under matched budgets. The memory-feasibility demonstration at 27B is compelling but rests on the absence of a feasible RL baseline rather than a head-to-head win. The open questions the paper leaves—population scaling laws with model capability, quantization-compatible ES infrastructure, continual-learning behavior of dense ES updates, and fully coupled skill–parameter co-evolution—are concrete and well-scoped follow-ups to its claims.