Agentic ESOpt: Evolutionary Fine-Tuning for LLM Agents

• Agentic ESOpt is a black-box evolution-strategy framework that perturbs full LLM parameters, evaluates complete tool-using trajectories, and updates the model using normalized reward-weighted directions.
• The method targets long-horizon tasks with branching interactions and sparse rewards, improving Sudoku performance to 53.13% at 15 turns and raising reported Math and DocVQA results by an average of 13.7 points over the base model.
• By storing perturbation seeds rather than activations, critics, reference models, or optimizer states, Agentic ESOpt enables inference-level GPU memory use and can co-evolve model parameters with prompts, skills, memories, or heuristics.

Agentic ESOpt is an evolution-strategy framework for fine-tuning long-horizon LLM agents with full-parameter updates using inference-level GPU memory. At each optimization iteration, it samples perturbations around the current LLM parameters, evaluates the resulting agents through complete environment trajectories, normalizes scalar rewards, and applies an online reward-weighted update. The framework is designed for agentic settings characterized by branching interactions, sparse or terminal rewards, tool use, and substantial horizon-dependent credit-assignment difficulty. Its broader design includes parameter–context co-evolution, in which LLM parameters and external prompts, skills, memories, or heuristics may be adapted together. The principal paper is “Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements” (Zheng et al., 18 Aug 2026).

1. Conceptual foundations and scope

Agentic ESOpt treats an LLM agent as a policy $\pi_\theta$ that repeatedly generates actions conditioned on interaction history and external context:

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$

where $\theta$ denotes the LLM parameter vector, $a_t$ is the action at turn $t$, $\boldsymbol{o}_{\leq t}$ is the observation history, and $c_t$ may contain prompts, skills, memories, or tool instructions. An episode produces a trajectory

$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$

with realized horizon $H$. The trajectory return is

$$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$

In many of the studied environments, intermediate rewards are zero and the useful signal is essentially a terminal success score. The optimization objective for fixed external context is

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$0

Agentic ESOpt is a zeroth-order or black-box optimization method. It does not differentiate through tokens, model activations, action probabilities, or environment transitions. Instead, it estimates parameter-space improvement from the rewards of perturbed agents.

The framework is distinct from several adjacent uses of “agentic optimization.” Primitive Agentic First-Order Optimization applies reinforcement learning to sequentially select among first-order update operators, rather than perturbing LLM parameters (Sala, 2024). EPOCH provides orchestration, evaluation governance, and artifact provenance for multi-round optimization, but does not define an evolutionary strategy (Liu et al., 10 Mar 2026). AEvo introduces meta-editing of the procedure or agent context that controls future evolution, rather than only updating the candidate artifact (Zhang et al., 13 May 2026). AgentOpt studies end-to-end configuration of LLM-agent pipelines using bandits, Bayesian optimization, and other black-box search methods, but does not implement an evolutionary strategy (Hua et al., 7 Apr 2026).

Agentic ESOpt is therefore specifically concerned with full-parameter evolutionary adaptation of an acting LLM policy under scalar trajectory feedback. It is not a population-based optimizer in the conventional sense of maintaining a persistent population of model copies: perturbation directions are sampled for an update population, evaluated independently, and then removed before the reward-weighted parameter update.

2. Motivation: long-horizon agents and reinforcement-learning limitations

The framework is motivated by four interacting limitations of conventional agentic RL.

Memory-intensive full-parameter training: Policy-gradient systems generally require model activations, rollout tokens and hidden states, gradients, optimizer states, and sometimes a reference model, critic, or value function. The paper reports that Qwen3.5-4B inference requires about $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$1 GB in its Sudoku setup, whereas corresponding full-parameter GRPO and PPO configurations require $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$2 GB and $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$3 GB, respectively.

Backpropagation and reference-model overhead: GRPO-like methods require policy computation, reference-policy computation, and backward propagation. PPO additionally requires critic computation. The paper gives approximate model-side FLOPs:

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$4

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$5

and

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$6

Here, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$7 is the number of update rounds, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$8 the prompt batch size, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$9 the number of rollouts or perturbation directions, $\theta$0 the number of model parameters, and $\theta$1 the average number of model-processed tokens per trajectory. These expressions are approximate and describe model-side computation rather than environment-execution cost.

Branching interactions: In a browser, tool-use, GUI, or coding environment, each action changes the next observation and therefore the future prompt. A mistaken click, tool call, navigation step, or intermediate computation may lead to a qualitatively different trajectory. The number of possible interaction paths grows rapidly with the horizon.

Sparse rewards and credit assignment: A terminal reward may provide little evidence about which action caused success or failure. GRPO effectively attaches a trajectory-level relative advantage to many action or token terms:

$\theta$2

A stylized policy-gradient estimator is

$\theta$3

where $\theta$4 is a baseline. Under simplifying assumptions in which the return is weakly correlated with individual actions and per-turn score terms are approximately uncorrelated, the paper gives the scaling argument

$\theta$5

This is a scaling argument rather than a claim that ES variance is independent of horizon. The distinction is that RL explicitly accumulates action-level score terms across turns, whereas ES attributes the complete return to a single coherent parameter perturbation.

3. Evolution-strategy objective and update mechanism

Agentic ESOpt optimizes the Gaussian-smoothed objective

$\theta$6

where $\theta$7 is the perturbation scale and $\theta$8 is a standard Gaussian direction in the full parameter space. A perturbed parameter vector is

$\theta$9

The ES score-function identity gives

$a_t$0

Expanding through environment trajectories,

$a_t$1

At iteration $a_t$2, the method samples $a_t$3 independent directions $a_t$4. For each direction, it temporarily applies $a_t$5, runs the perturbed agent, records a complete trajectory and scalar reward $a_t$6, and then reverts the perturbation. The canonical Monte Carlo estimator is

$a_t$7

with

$a_t$8

A perturbation-independent baseline may replace $a_t$9 by $t$0, because $t$1.

In the implemented update, population rewards are standardized:

$t$2

where

$t$3

and

$t$4

The online reward-weighted parameter update is

$t$5

Unlike the canonical estimator, this implementation omits the explicit $t$6 factor. Consequently, $t$7, rather than $t$8, controls the practical update magnitude. High-reward perturbations receive positive weight, low-reward perturbations receive negative weight, and the model moves toward parameter-space directions associated with better complete trajectories.

The method stores random seeds for perturbation directions instead of materializing a full $t$9-dimensional noise vector for every candidate. A direction can be regenerated from its seed, applied in place,

$\boldsymbol{o}_{\leq t}$0

and removed after evaluation,

$\boldsymbol{o}_{\leq t}$1

The same seed is replayed when constructing the reward-weighted update. This avoids storing rollout activations and large populations of model copies.

4. Perturbation scheduling and parameter–context co-evolution

Agentic ESOpt uses cosine decay for the perturbation scale:

$\boldsymbol{o}_{\leq t}$2

Here, $\boldsymbol{o}_{\leq t}$3 is the initial perturbation radius, $\boldsymbol{o}_{\leq t}$4 the final radius, $\boldsymbol{o}_{\leq t}$5 the total number of update steps, and $\boldsymbol{o}_{\leq t}$6 the current step. Early iterations use a larger neighborhood for exploration and later iterations use smaller perturbations for local refinement.

The Gaussian-smoothing expansion is

$\boldsymbol{o}_{\leq t}$7

A nonzero $\boldsymbol{o}_{\leq t}$8 therefore optimizes neighborhood behavior rather than exactly the un-smoothed objective. It can discourage sharp local optima, while also introducing smoothing bias. For train-time fine-tuning, the authors generally retain a nonzero $\boldsymbol{o}_{\leq t}$9; for test-time compute, where the goal is the best solution for the current task rather than generalization, $c_t$0 is decayed to zero.

The framework can optimize both LLM parameters and external context. Let $c_t$1 denote trajectories and scores collected at iteration $c_t$2, $c_t$3 the ES update, and $c_t$4 a context-update rule:

$c_t$5

$c_t$6

The external context may include prompts, skill documents, memories, heuristics, or tool instructions. Parameter optimization changes the underlying policy, while context optimization changes the behavior induced by that policy.

In Trace2Skill experiments, the method first adapts the model using No Skill trajectories, after which those trajectories are passed to a skill-distillation procedure. The resulting system combines ES-adapted parameters with a skill document distilled from trajectories. In WebArena, this combination is sequential rather than fully alternating: a 70-generation parameter run is followed by one post-hoc skill-distillation stage.

In automatic heuristic design, an LLM proposes Python heuristics and an external solver evaluates the resulting algorithm. Agentic ESOpt adapts the proposal distribution while Sample or EoH continues to evolve heuristic candidates. For EoH, the original evolutionary operators are preserved and ES updates are attached to mutation operators $c_t$7 and $c_t$8. For Sample, batches of proposals form ES update populations. Thus the coupled search acts over heuristic space and LLM parameter space, with the latter indirectly influencing solver behavior.

5. Evaluation environments and empirical results

Agentic ESOpt has been evaluated in controlled Sudoku, mathematical reasoning, DocVQA, WebArena-Lite, and automatic heuristic-design settings (Zheng et al., 18 Aug 2026).

Sudoku

The Sudoku environment requires one action per masked cell and supplies essentially terminal success feedback. The minimum successful horizon is

$c_t$9

Masking 5, 10, or 15 cells gives $\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$0. The setup uses Qwen3.5-4B, four H100 80GB GPUs, 32 training puzzles, and 32 evaluation puzzles per horizon.

Method	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$1	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$2	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$3
PPO	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$4	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$5	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$6
Stronger GRPO	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$7	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$8	$\boldsymbol{\tau}=(o_1,a_1,\ldots,o_H,a_H),$9
Agentic ESOpt	$H$0	$H$1	$H$2

The results show a horizon-dependent crossover rather than universal ES superiority. PPO is strongest at five turns, GRPO at ten, and Agentic ESOpt at fifteen, where it exceeds GRPO by $H$3 percentage points.

At $H$4, Agentic ESOpt reaches $H$5, Vanilla ES without sigma decay reaches $H$6, and a train-time terminal sigma of zero reaches $H$7. These comparisons support the use of broad early exploration followed by smaller but nonzero perturbations.

Mathematical reasoning and DocVQA

The Math experiments use 400 DAPO training problems, 100 held-out DAPO problems, 30 AIME 2026 problems, up to 50 interaction turns, Python/bash tool use, and exact-match final-answer reward. DocVQA uses a 50-question training subset, 100 held-out questions, up to 50 turns, image inspection and OCR tools, ANLS, and thresholded accuracy.

With Qwen3.5-4B, $H$8, $H$9, and cosine decay from $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$0 to $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$1, Agentic ESOpt without skills improves over the No Skill baseline by:

• $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$2 points on DAPO Mean@4;
• $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$3 points on AIME Mean@4;
• $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$4 points on DocVQA Mean@4 accuracy.

Averaged across these three metrics, the paper reports a $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$5-point improvement over the base model and an $$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$6-point improvement over Agentic GRPO. The method also composes with Trace2Skill, producing the strongest Qwen3.5-4B Mean@4 results on the reported Math and DocVQA metrics.

WebArena-Lite

WebArena-Lite contains 165 browser tasks across Reddit, GitLab, CMS, Map, OSS, and Wikipedia. The agent receives WebRL-style textual browser observations and performs id-based browser actions, with binary task success as reward.

The experiment uses Qwen3.5-27B, four H100 80GB GPUs, eight perturbation directions per update, eight web tasks per direction, 70 ES generations, a maximum of 30 browser actions per task, and a 2,048-token response budget.

System	WebArena-Lite success
No Skill	$$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$7
Trace2Skill	$$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$8
Agentic ESOpt	$$R(\boldsymbol{\tau})=\sum_{t=1}^{H}\gamma^{t-1}r_t.$$9
Agentic ESOpt + Trace2Skill	$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$00

Full-parameter Agentic ESOpt improves the No Skill baseline by $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$01 percentage points. The result is primarily a feasibility demonstration for full-parameter adaptation of a 27B model with inference-level memory requirements. It is not a matched-hardware performance comparison against full-parameter RL, because full-parameter RL was not practical in that setting.

Automatic heuristic design

The automatic heuristic-design experiments use Llama-3.1-8B-Instruct on eight RTX 3090 GPUs. They cover constructive TSP, Knapsack, Admissible Set Problem, ACO-style TSP, CVRP, and Bin Packing, with evaluation budgets $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$02 and $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$03, and Sample and EoH as outer search baselines.

For constructive tasks, the normalized optimality gap is

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$04

and the gain over baseline $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$05 is

$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$06

Agentic ESOpt improves 28 of 36 matched method–budget comparisons. For constructive settings, Agentic ESOpt plus EoH improves all six test sets at both budgets, Agentic ESOpt plus Sample improves 9 of 12 comparisons, and the combined constructive comparisons improve 21 of 24 times.

Removing the reward-weighted ES update weakens results, as does removing cosine decay. Retuning EoH’s LLM sampling temperature does not reproduce the gain. The on-the-fly runtime overhead for Agentic ESOpt plus Sample is approximately $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$07–$$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$08.

6. Computational properties, limitations, and research significance

Agentic ESOpt’s principal systems claim is that full-parameter optimization requires approximately inference-level GPU memory. Each candidate requires a forward rollout, while the described update requires no policy-gradient computation, retained activation graph, critic, reference model, or optimizer moments. The method moves computational cost from backward propagation and training-state storage to independent forward rollouts and environment evaluations.

This trade-off is favorable when model memory is the binding constraint and environment evaluations are sufficiently affordable. It is less favorable when browser execution, simulation, solver execution, or API access dominates cost, because ES may require more independent trajectories than RL. In Sudoku, the method uses 32 ES directions versus eight GRPO rollouts; the lower per-trajectory model-side cost approximately offsets the larger ES population in the reported FLOPs comparison.

The framework is not universally superior to policy-gradient RL. The Sudoku crossover demonstrates that PPO and GRPO can outperform Agentic ESOpt at shorter horizons. ES remains sensitive to perturbation scale, population size, reward sparsity, parameter-space geometry, and task distribution. A population-sensitivity study reports that Qwen3.5-4B benefits substantially from increasing $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$09 from 8 to 16 on 15-turn Sudoku, whereas Qwen3.5-9B is less sensitive.

Continual-learning behavior remains unresolved. Dense ES updates could induce a random walk in directions unrelated to the objective, potentially damaging previously acquired capabilities. In the WebArena analysis, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$10 of parameter changes are at most $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$11, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$12 are at most the perturbation scale $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$13, and $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$14 are below $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$15. These measurements characterize the reported runs but do not establish long-term preservation of capabilities.

Other limitations include the following:

• Sparse or nondiscriminative rewards: ES can still fail when few perturbations produce informative outcomes.
• Expensive environments: More forward rollouts may be costly even when GPU memory is reduced.
• Hyperparameter sensitivity: $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$16, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$17, $$a_t\sim\pi_\theta(a_t\mid \boldsymbol{o}_{\leq t},c_t),$$18, and the terminal perturbation scale may depend on model and task.
• Co-evolution incompleteness: The reported WebArena parameter–skill combination is sequential rather than fully alternating.
• Quantization: Dense perturbations of quantized weights require scale-aware noise and numerically stable seed replay.
• Generalization: Scaling to frontier proprietary models and establishing population-scaling laws remain open.
• Evaluation scope: The reported evidence covers selected reasoning, browser, document, Sudoku, and heuristic-design tasks rather than a broad deployment distribution.

Agentic ESOpt is complementary to other agentic optimization architectures. AEPO balances entropy during rollout and policy updating for multi-turn tool-use RL, but it remains a gradient-based PPO/GRPO-like method rather than an evolutionary strategy (Dong et al., 16 Oct 2025). ODYSSE’s ESPO introduces episode-level rewards and episodic advantage estimation for personalized GUI reasoning, but also remains a policy-gradient method (Beheshti et al., 14 Jul 2026). AgentOpt provides a discrete, end-to-end evaluation substrate for optimizing model assignments across agent-pipeline roles, including cost and latency, and suggests evolutionary extensions for highly epistatic pipeline configurations (Hua et al., 7 Apr 2026). AgentOptics demonstrates protocol-grounded, closed-loop black-box control over optical hardware, but its reported search procedures are finite-grid, threshold, or coordinate-wise methods rather than ES (Wang et al., 23 Feb 2026). Physically Constrained Agentic AI for Energy Scheduling similarly separates LLM proposal from deterministic physical authorization in MIP and MILP scheduling (Zhao et al., 10 Aug 2026).

The central significance of Agentic ESOpt is the alignment between parameter-space perturbation and long-horizon agent behavior. Rather than assigning a sparse terminal outcome to individual token or turn-level decisions, it evaluates whether a nearby version of the entire agent produces a better trajectory. This provides:

1. Model scalability through full-parameter adaptation with inference-level memory.
2. Black-box flexibility through a scalar feedback interface compatible with tools, browsers, code execution, solvers, prompts, skills, and heuristic search.
3. Long-horizon attribution through reward-weighted parameter perturbations without explicit accumulation of action-score terms over the trajectory.
4. Context compatibility through parameter–context co-evolution.
5. Search integration through insertion into existing heuristic and evolutionary procedures.

The resulting method should not be interpreted as a universal replacement for RL or classical evolutionary optimization. Its strongest applicability is suggested for settings with sparse feedback, long and branching trajectories, large models whose full-parameter RL training exceeds available memory, and reliable scalar evaluation. Its empirical results support a regime-dependent advantage, particularly at longer horizons and in full-parameter adaptation, while leaving open questions concerning environment cost, continual learning, perturbation scaling, co-evolution, robustness, and formal convergence.