Relative Experience Optimization in RL

• Relative Experience Optimization is defined as a method that prioritizes replay based on an upper-bound of the TD error, modulated by on-policyness.
• It optimizes the extraction and reuse of experiences by quantifying their marginal utility for policy improvement, transfer, and variance reduction.
• Empirical studies on benchmarks like grid-world and CartPole show that REO accelerates convergence and yields higher cumulative rewards compared to uniform replay.

Relative Experience Optimization is a label applied to multiple but related constructions in reinforcement learning and experience-based optimization. In the most specific formulation, it denotes a replay-priority scheme derived from theoretical bounds on the value of replaying a transition once, replacing plain $|\text{TD}|$ prioritization by an upper bound of the form $\rho_{\max}|\text{TD}|$ (Li et al., 2021). In broader usage, the same phrase or closely aligned ideas refer to optimizing the extraction, selection, reuse, or conditioning of experience by its relative usefulness to the current policy, to downstream transfer, or to gradient-variance reduction rather than by uniform reuse alone (Fan et al., 11 May 2026).

1. Value of experience as the foundational quantity

In the value-based replay formulation, the “value of an experience” is defined in an economic sense as the increase in expected cumulative reward that a rational, fully informed agent would obtain by replaying a stored transition once. Let $e_k = (s_k, a_k, r_k, s_k')$, and let $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$ denote the policy, state-value, and action-value before the backup, while $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$ denote the corresponding quantities immediately after exactly one update. Three metrics are introduced (Li et al., 2021):

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$

$$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$

These quantities satisfy

$\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$

This decomposition separates the gain attributable to policy change from the gain attributable to value evaluation. Its significance is that replay is no longer treated as a heuristic memory mechanism; it is given a utility-theoretic interpretation in terms of value added to cumulative reward. A plausible implication is that replay priorities can be derived from value bounds rather than chosen solely by empirical convention.

2. Q-learning bounds and the reinterpretation of TD error

For tabular Q-learning, a single backup over $e_k$ is

$\rho_{\max}|\text{TD}|$0

with

$\rho_{\max}|\text{TD}|$1

Under greedy improvement, $\rho_{\max}|\text{TD}|$2. The central theoretical result is Theorem 1:

$\rho_{\max}|\text{TD}|$3

The proof sketch relies on the contraction property of $\rho_{\max}|\text{TD}|$4 and on the fact that only $\rho_{\max}|\text{TD}|$5 changes, by at most $\rho_{\max}|\text{TD}|$6. For $\rho_{\max}|\text{TD}|$7, the change is $\rho_{\max}|\text{TD}|$8 if $\rho_{\max}|\text{TD}|$9, and zero otherwise. Empirically, in a $e_k = (s_k, a_k, r_k, s_k')$0 grid-world maze and in CartPole, plots of the three value metrics against $e_k = (s_k, a_k, r_k, s_k')$1 place almost all points under the identity line, with many lying tightly on it (Li et al., 2021).

The importance of this result is not that $e_k = (s_k, a_k, r_k, s_k')$2 equals value of experience, but that it upper-bounds it. This addresses a common misconception in prioritized replay: large TD error is not itself the value added by replay, but it can serve as a theoretically justified proxy for an upper bound on that value.

3. Maximum-entropy extension, on-policyness, and the REO priority

In maximum-entropy RL, rewards are augmented by $e_k = (s_k, a_k, r_k, s_k')$3, the soft Bellman backup is

$e_k = (s_k, a_k, r_k, s_k')$4

and the new policy is

$e_k = (s_k, a_k, r_k, s_k')$5

The soft TD error is

$e_k = (s_k, a_k, r_k, s_k')$6

The same value metrics are defined in entropy-augmented form. Theorem 2 states that if

$e_k = (s_k, a_k, r_k, s_k')$7

then for each $e_k = (s_k, a_k, r_k, s_k')$8,

$e_k = (s_k, a_k, r_k, s_k')$9

Theorem 3 gives a lower bound for $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$0 and $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$1: $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$2 and for $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$3,

$\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$4

No analogous uniform lower bound holds for $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$5. In words, each soft-value metric lies in the interval

$\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$6

where $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$7 captures how “on-policy” the experience is. Empirical plots in soft Q-learning on the grid-world maze and CartPole show samples clustering between the two bounding lines (Li et al., 2021).

This leads directly to Relative Experience Optimization. The replay priority is set by the estimated upper bound: $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$8 Operationally, one defines

$\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$9

where $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$0 estimates $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$1, samples mini-batches proportionally to $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$2, and applies importance-sampling corrections as in PER. At storage time, $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$3 is recorded; at replay time, $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$4 is recomputed by a forward pass of the Q-network; then

$\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$5

This criterion carries over to standard Q-learning by setting $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$6 (Li et al., 2021).

4. Replay optimization beyond the original REO derivation

Several related frameworks optimize experience replay without using the specific $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$7 construction. In Experience Replay Optimization (ERO), replay is cast as a separate policy-gradient problem with two interacting policies: an agent policy $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$8 and a replay policy $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$9. The replay policy takes a feature vector $$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$0 for each stored transition, outputs a priority score $$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$1, defines a Bernoulli sampling distribution over the buffer, and is trained to maximize the “replay reward”

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$2

the change in the agent’s cumulative return after training on the sampled subset. The replay-policy gradient is derived by REINFORCE, using

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$3

with

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$4

The reported behavior is adaptive rather than rule-based: the learned replay policy tends to prefer more recent transitions and, in some tasks, slightly lower TD-error examples (Zha et al., 2019).

In multi-agent reinforcement learning, MAC-PO formulates prioritized replay as regret minimization over sampling weights of transitions. At iteration $$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$5, nonnegative weights $$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$6 satisfy

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$7

and the outer objective minimizes policy regret subject to a weighted Bellman-projection constraint. After relaxation and a Lagrangian treatment, the closed-form optimal weights are proportional to

$$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$8

up to normalization and a negligible revisit error $$\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),$$9. The factors were interpreted as Bellman error, value-enhancement, on-policity, and a purely multi-agent joint-action term (Mei et al., 2023).

Variance Reduction Experience Replay (VRER) addresses a different criterion: reducing the variance of policy-gradient estimators when reusing historical trajectories. It forms importance-sampling weights

$$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$0

and optimizes a replay-weighted surrogate objective over fresh and reused trajectories. A central insight is a bias-variance trade-off: reusing older experience increases bias but reduces gradient variance. The more recent analysis states the informal objective as minimizing

$$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$1

and proposes a variance-ratio test or a KL-based approximation to select which past policies enter the reuse set $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$2 (Zheng et al., 2021, Zheng et al., 5 Feb 2026).

Taken together, these formulations show that “experience optimization” may target distinct objective functions: cumulative-return improvement, regret minimization, or gradient-variance reduction. This suggests that the relative worth of experience is architecture- and objective-dependent rather than universal.

5. Experience optimization in self-evolving agents and cross-task transfer

In Evolving-RL, “relative experience optimization” appears in a broader form: a shared policy $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$3 acts as both extractor and solver, and experience extraction and utilization are jointly optimized within the same parameter vector $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$4. For a completed source interaction with task description $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$5, trajectory $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$6, and source reward $$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$7, the extractor samples candidate skills

$$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$8

For each source task, the method retrieves

$$\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),$$9

evaluates each skill on each retrieved task, and defines the skill reward

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$0

The extractor advantage is

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$1

while the solver advantage is

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$2

Two PPO-style losses,

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$3

are combined as

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$4

The conceptual claim is explicit: the method optimizes for “relative” usefulness of experience by tying extractor rewards directly to cross-task downstream gains and by simultaneously refining a solver to exploit informative skills and ignore spurious ones (Fan et al., 11 May 2026).

Beyond RL replay, experience-based expensive optimization treats each expensive black-box problem as part of a family of related tasks. Shared task-independent parameters are meta-learned for a deep kernel learning surrogate and then adapted to a target task using only $$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$5 solutions from the target task, with $$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$6 the dimension of the decision space. The learned “experience” becomes the initialization of the surrogate, enabling regression-based surrogate-assisted evolutionary algorithms to start from $$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$7 evaluations instead of the usual $$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$8 or more (Yu et al., 2023).

A related but training-free line appears in Training-Free GRPO. Vanilla GRPO uses normalized numeric advantage

$$\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}), \quad a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).$$9

whereas Training-Free GRPO replaces it by a textual “semantic advantage” $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$0 distilled from grouped rollouts and accumulated into an experience library $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$1. At inference time, $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$2 is inserted into the prompt as a learned token prior: $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$3 No parameter update is performed. This is not replay prioritization in the strict Q-learning sense, but it preserves the central theme of relative experience evaluation: experience is ranked, distilled, and re-injected according to its group-relative usefulness (Cai et al., 9 Oct 2025).

6. Empirical findings, limitations, and recurring misconceptions

The original replay-priority formulation was implemented in “soft DQN” on nine Atari games. The reported baselines were uniform replay, PER with priority $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$4, and VER with priority $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$5. All agents trained for $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$6 frames with the Mnih et al. network, $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$7, $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$8, buffer $\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.$9, and batch $e_k$0. Uniform replay provided the baseline; PER yielded $e_k$1 average gain over uniform; VER yielded $e_k$2 gain over uniform and $e_k$3 gain over PER, and typically converged faster (Li et al., 2021).

Other experience-optimization schemes report gains under different objectives. MAC-PO converges faster and attains $e_k$4 capture rate on Predator-Prey under both no-punishment and heavy-punishment settings, yields $e_k$5–$e_k$6 pp higher win-rates than uniform replay, PER, DisCor, and ReMERN on SMAC micromanagement, and improves super-hard MMM2 by $e_k$7–$e_k$8 pp over several MARL baselines; disabling the joint-action term drops final win-rate by $e_k$9 pp (Mei et al., 2023). ERO adds only $\rho_{\max}|\text{TD}|$00–$\rho_{\max}|\text{TD}|$01 extra CPU/GPU time and substantially accelerates learning on eight continuous-control benchmarks, while PER gives mixed or negligible gains for DDPG on these tasks (Zha et al., 2019). VRER reports average-return improvements such as $\rho_{\max}|\text{TD}|$02 for PPO-VRER on CartPole, $\rho_{\max}|\text{TD}|$03 for TRPO-VRER on Hopper, and up to $\rho_{\max}|\text{TD}|$04 for A2C-VRER on InvertedPendulum, alongside reported gradient-variance reductions of $\rho_{\max}|\text{TD}|$05–$\rho_{\max}|\text{TD}|$06 or $\rho_{\max}|\text{TD}|$07–$\rho_{\max}|\text{TD}|$08 depending on the experiment (Zheng et al., 2021, Zheng et al., 5 Feb 2026).

In self-evolving language-agent settings, Evolving-RL reports on ALFWorld unseen tasks: GRPO $\rho_{\max}|\text{TD}|$09 at $\rho_{\max}|\text{TD}|$10 success, Evolving-RL $\rho_{\max}|\text{TD}|$11 at $\rho_{\max}|\text{TD}|$12 success, and Evolving-RL $\rho_{\max}|\text{TD}|$13 at $\rho_{\max}|\text{TD}|$14 success; on Mind2Web overall action accuracy it reports $\rho_{\max}|\text{TD}|$15 for GRPO $\rho_{\max}|\text{TD}|$16, $\rho_{\max}|\text{TD}|$17 for Evolving-RL $\rho_{\max}|\text{TD}|$18, and $\rho_{\max}|\text{TD}|$19 for Evolving-RL $\rho_{\max}|\text{TD}|$20. The ALFWorld ablation further shows that solver-only improves unseen tasks to $\rho_{\max}|\text{TD}|$21, extractor-only reaches $\rho_{\max}|\text{TD}|$22 on seen tasks but only $\rho_{\max}|\text{TD}|$23 on unseen tasks, and full co-evolution reaches $\rho_{\max}|\text{TD}|$24 on seen tasks and $\rho_{\max}|\text{TD}|$25 on unseen tasks (Fan et al., 11 May 2026). Training-Free GRPO reports AIME24/AIME25 gains from $\rho_{\max}|\text{TD}|$26 to $\rho_{\max}|\text{TD}|$27 without tools, gains from $\rho_{\max}|\text{TD}|$28 to $\rho_{\max}|\text{TD}|$29 with ReAct and a code interpreter, and a WebWalkerQA pass@1 increase from $\rho_{\max}|\text{TD}|$30 to $\rho_{\max}|\text{TD}|$31, with $\rho_{\max}|\text{TD}|$32 distilled in $\rho_{\max}|\text{TD}|$33 epochs at cost %%%%100$\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$0101%%%%d$\rho_{\max}|\text{TD}|$3690$$fewer initialization runs across DTLZ1–DTLZ6 (<a href="/papers/2304.04166" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Yu et al., 2023</a>).</p> <p>Several misconceptions recur across this literature. One is that experience value is equivalent to TD error; the value-perspective results show an upper-bound relation, not an identity, and in soft Q-learning the bound is modulated by on-policyness, with no analogous uniform lower bound for$$\rho_{\max}|\text{TD}|$37 (Li et al., 2021). A second is that more historical reuse is always beneficial; VRER’s theoretical contribution is precisely to formalize the bias-variance trade-off, where older samples can reduce variance while increasing bias (Zheng et al., 2021, Zheng et al., 5 Feb 2026). A third is that experience optimization is only a replay-buffer problem; Evolving-RL, experience-based surrogate adaptation, and Training-Free GRPO show that experience can also be extracted as skills, internalized into model parameters, or distilled into a token prior (Fan et al., 11 May 2026, Yu et al., 2023, Cai et al., 9 Oct 2025).

The cumulative picture is therefore heterogeneous but coherent. Relative Experience Optimization, in its narrow sense, is the use of $\rho_{\max}|\text{TD}|$38 as an upper-bound priority for replay. In a broader sense, it names a research direction in which the usefulness of experience is evaluated relative to current policy, downstream transfer benefit, coordination structure, or estimator variance. This suggests that the unifying question is not whether experience should be reused, but how its marginal utility should be formalized for the learning system at hand.