---
title: Relative Experience Optimization in RL
url: https://www.emergentmind.com/topics/relative-experience-optimization
type: topic
---

# Relative Experience Optimization in RL

Relative Experience Optimization is a label applied to multiple but related constructions in reinforcement learning and experience-based optimization. In the most specific formulation, it denotes a replay-priority scheme derived from theoretical bounds on the value of replaying a transition once, replacing plain $|\text{TD}|$ prioritization by an upper bound of the form $\rho_{\max}|\text{TD}|$ [2102.03261]. In broader usage, the same phrase or closely aligned ideas refer to optimizing the extraction, selection, reuse, or conditioning of experience by its relative usefulness to the current policy, to downstream transfer, or to gradient-variance reduction rather than by uniform reuse alone [2605.10663].

## 1. Value of experience as the foundational quantity

In the value-based replay formulation, the “value of an experience” is defined in an economic sense as the increase in expected cumulative reward that a rational, fully informed agent would obtain by replaying a stored transition once. Let
$e_k = (s_k, a_k, r_k, s_k')$,
and let $\pi_{\text{old}}, Q_{\text{old}}, V_{\text{old}}$ denote the policy, state-value, and action-value before the backup, while $\pi_{\text{new}}, Q_{\text{new}}, V_{\text{new}}$ denote the corresponding quantities immediately after exactly one update. Three metrics are introduced [2102.03261]:

\[
\mathrm{EVB}(e_k) \coloneqq V_{\pi_{\text{new}}}(s_k) - V_{\pi_{\text{old}}}(s_k),
\]

\[
\mathrm{PIV}(e_k) \coloneqq \sum_a \bigl[\pi_{\text{new}}(a|s_k) - \pi_{\text{old}}(a|s_k)\bigr] Q_{\text{new}}(s_k,a),
\]

\[
\mathrm{EIV}(e_k) \coloneqq Q_{\text{new}}(s_k,a_{\text{old}}) - Q_{\text{old}}(s_k,a_{\text{old}}),
\quad
a_{\text{old}} = \arg\max_a Q_{\text{old}}(s_k,a).
\]

These quantities satisfy
\[
\mathrm{EVB} = \mathrm{PIV} + \mathrm{EIV}.
\]

This decomposition separates the gain attributable to policy change from the gain attributable to value evaluation. Its significance is that replay is no longer treated as a heuristic memory mechanism; it is given a utility-theoretic interpretation in terms of value added to cumulative reward. A plausible implication is that replay priorities can be derived from value bounds rather than chosen solely by empirical convention.

## 2. Q-learning bounds and the reinterpretation of TD error

For tabular Q-learning, a single backup over $e_k$ is
\[
Q_{\text{new}}(s_k,a_k) = Q_{\text{old}}(s_k,a_k) + \alpha\,\mathrm{TD},
\]
with
\[
\mathrm{TD} \coloneqq r_k + \gamma \max_{a'} Q_{\text{old}}(s_k',a') - Q_{\text{old}}(s_k,a_k).
\]

Under greedy improvement, $\pi_{\text{new}}(s) = \arg\max_a Q_{\text{new}}(s,a)$. The central theoretical result is Theorem 1:

\[
|\mathrm{EVB}(e_k)| \le \alpha |\mathrm{TD}|,
\quad
|\mathrm{PIV}(e_k)| \le \alpha |\mathrm{TD}|,
\quad
|\mathrm{EIV}(e_k)| \le \alpha |\mathrm{TD}|.
\]

The proof sketch relies on the contraction property of $\max$ and on the fact that only $Q(s_k,a_k)$ changes, by at most $\alpha|\mathrm{TD}|$. For $\mathrm{EIV}$, the change is $\alpha\,\mathrm{TD}$ if $a_{\text{old}}=a_k$, and zero otherwise. Empirically, in a $5\times 5$ grid-world maze and in CartPole, plots of the three value metrics against $|\mathrm{TD}|$ place almost all points under the identity line, with many lying tightly on it [2102.03261].

The importance of this result is not that $|\mathrm{TD}|$ equals value of experience, but that it upper-bounds it. This addresses a common misconception in prioritized replay: large TD error is not itself the value added by replay, but it can serve as a theoretically justified proxy for an upper bound on that value.

## 3. Maximum-entropy extension, on-policyness, and the REO priority

In maximum-entropy RL, rewards are augmented by $+\beta H(\pi(\cdot|s))$, the soft Bellman backup is
\[
Q_{\text{new}}(s,a)=r + \gamma V_{\text{old}}(s'),
\qquad
V(s)=\beta \log \sum_a \exp[Q(s,a)/\beta],
\]
and the new policy is
\[
\pi_{\text{new}}(a|s)\propto \exp[Q_{\text{new}}(s,a)/\beta].
\]

The soft TD error is
\[
\mathrm{TD}^{\text{soft}} \coloneqq r + \gamma V_{\text{old}}(s') - Q_{\text{old}}(s,a).
\]

The same value metrics are defined in entropy-augmented form. Theorem 2 states that if
\[
\rho_{\max} = \max\{\pi_{\text{old}}(a_k|s_k),\,\pi_{\text{new}}(a_k|s_k)\},
\]
then for each $M \in \{\mathrm{EVB}, \mathrm{PIV}, \mathrm{EIV}\}$,
\[
|M^{\text{soft}}(e_k)| \le \rho_{\max} |\mathrm{TD}^{\text{soft}}|.
\]

Theorem 3 gives a lower bound for $\mathrm{EVB}^{\text{soft}}$ and $\mathrm{EIV}^{\text{soft}}$:
\[
\rho_{\min} = \min\{\pi_{\text{old}}(a_k|s_k),\,\pi_{\text{new}}(a_k|s_k)\},
\]
and for $M\in\{\mathrm{EVB},\mathrm{EIV}\}$,
\[
|M^{\text{soft}}(e_k)| \ge \rho_{\min} |\mathrm{TD}^{\text{soft}}|.
\]

No analogous uniform lower bound holds for $\mathrm{PIV}^{\text{soft}}$. In words, each soft-value metric lies in the interval
\[
[\rho_{\min},\,\rho_{\max}] \cdot |\mathrm{TD}^{\text{soft}}|,
\]
where $\rho$ captures how “on-policy” the experience is. Empirical plots in soft Q-learning on the grid-world maze and CartPole show samples clustering between the two bounding lines [2102.03261].

This leads directly to Relative Experience Optimization. The replay priority is set by the estimated upper bound:
\[
\mathrm{priority}(e) = \rho_{\max}(e)\,|\mathrm{TD}(e)|.
\]
Operationally, one defines
\[
p_i = |\mathrm{TD}_i| \cdot \hat{\rho}_i,
\]
where $\hat{\rho}_i$ estimates $\max\{\pi_{\text{old}}(a_i|s_i), \pi_{\text{current}}(a_i|s_i)\}$, samples mini-batches proportionally to $p_i^\alpha$, and applies importance-sampling corrections as in PER. At storage time, $\pi_{\text{store}}=\pi_{\text{old}}(a|s)$ is recorded; at replay time, $\pi_{\text{current}}(a|s)$ is recomputed by a forward pass of the Q-network; then
\[
\hat{\rho} = \max\{\pi_{\text{store}},\,\pi_{\text{current}}\}.
\]
This criterion carries over to standard Q-learning by setting $\rho_{\max}\equiv 1$ [2102.03261].

## 4. Replay optimization beyond the original REO derivation

Several related frameworks optimize experience replay without using the specific $\rho_{\max}|\mathrm{TD}|$ construction. In Experience Replay Optimization (ERO), replay is cast as a separate policy-gradient problem with two interacting policies: an agent policy $\pi(a\mid s;\theta^\pi)$ and a replay policy $\mu \equiv \phi(f_i;\theta^\phi)$. The replay policy takes a feature vector $f_i$ for each stored transition, outputs a priority score $\lambda_i=\phi(f_i\mid\theta^\phi)\in(0,1)$, defines a Bernoulli sampling distribution over the buffer, and is trained to maximize the “replay reward”
\[
r^r = r_\pi^c - r_{\pi'}^c,
\]
the change in the agent’s cumulative return after training on the sampled subset. The replay-policy gradient is derived by REINFORCE, using
\[
\nabla_{\theta^\phi}J
= \mathbb{E}_I\bigl[r^r \nabla_{\theta^\phi}\log P(I\mid \theta^\phi)\bigr]
\]
with
\[
P(I\mid\theta^\phi)=\prod_{i=1}^N \phi(f_i)^{I_i}(1-\phi(f_i))^{1-I_i}.
\]
The reported behavior is adaptive rather than rule-based: the learned replay policy tends to prefer more recent transitions and, in some tasks, slightly lower TD-error examples [1906.08387].

In multi-agent reinforcement learning, MAC-PO formulates prioritized replay as regret minimization over sampling weights of transitions. At iteration $k$, nonnegative weights $w_k(s,\mathbf{u})$ satisfy
\[
\sum_{s,\mathbf{u}} \mu(s,\mathbf{u})\,w_k(s,\mathbf{u}) = 1,
\]
and the outer objective minimizes policy regret subject to a weighted Bellman-projection constraint. After relaxation and a Lagrangian treatment, the closed-form optimal weights are proportional to
\[
d^{\pi_k}(s,\mathbf u)\,
\bigl|Q_k-\mathcal B^*Q_{k-1}\bigr|\,
e^{-\lvert Q_k-Q^*\rvert}
\Bigl[
1+\sum_{i=1}^n \prod_{j\ne i}\pi_k^j(s,u^j) - n\prod_{i=1}^n\pi_k^i(s,u^i)
\Bigr],
\]
up to normalization and a negligible revisit error $\epsilon_k$. The factors were interpreted as Bellman error, value-enhancement, on-policity, and a purely multi-agent joint-action term [2302.10418].

Variance Reduction Experience Replay (VRER) addresses a different criterion: reducing the variance of policy-gradient estimators when reusing historical trajectories. It forms importance-sampling weights
\[
w(\tau;\theta,\theta_{\text{old}})
= \prod_{t=0}^{T-1}\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)},
\]
and optimizes a replay-weighted surrogate objective over fresh and reused trajectories. A central insight is a bias-variance trade-off: reusing older experience increases bias but reduces gradient variance. The more recent analysis states the informal objective as minimizing
\[
\mathrm{Var}\bigl(\widehat{\nabla}J_k^R\bigr)
\quad\text{subject to a small bias penalty},
\]
and proposes a variance-ratio test or a KL-based approximation to select which past policies enter the reuse set $\mathcal U_k$ [2110.08902, 2602.05379].

Taken together, these formulations show that “experience optimization” may target distinct objective functions: cumulative-return improvement, regret minimization, or gradient-variance reduction. This suggests that the relative worth of experience is architecture- and objective-dependent rather than universal.

## 5. Experience optimization in self-evolving agents and cross-task transfer

In Evolving-RL, “relative experience optimization” appears in a broader form: a shared policy $\pi_\theta$ acts as both extractor and solver, and experience extraction and utilization are jointly optimized within the same parameter vector $\theta$. For a completed source interaction with task description $x^{\text{src}}$, trajectory $\tau\sim\pi_\theta(\cdot\mid x^{\text{src}})$, and source reward $r^{\text{src}}$, the extractor samples candidate skills
\[
\{e_i\}_{i=1}^N \sim \pi_\theta(\cdot\mid s^e),
\qquad
s^e=(x^{\rm src},\,\tau,\,r^{\rm src}).
\]
For each source task, the method retrieves
\[
\mathcal X^{\rm ret} = \mathrm{TopK}_{x\in\mathcal D}\; s\bigl(\phi(x^{\rm src}),\phi(x)\bigr),
\qquad
s=\cosine,
\]
evaluates each skill on each retrieved task, and defines the skill reward
\[
R_i^e = \frac{1}{K}\sum_{j=1}^K r_{ij}.
\]
The extractor advantage is
\[
A_i^e
=
\frac{R_i^e - \mu(\{R_{i'}^e\})}{\sigma(\{R_{i'}^e\})},
\]
while the solver advantage is
\[
A_{ij}^s
=
\frac{r_{ij}-\mu(\{r_{i'j}\})}{\sigma(\{r_{i'j}\})}.
\]
Two PPO-style losses,
\[
\mathcal L_e(\theta)
\quad\text{and}\quad
\mathcal L_s(\theta),
\]
are combined as
\[
\mathcal L(\theta)=\lambda_e\mathcal L_e(\theta)+\lambda_s\mathcal L_s(\theta).
\]
The conceptual claim is explicit: the method optimizes for “relative” usefulness of experience by tying extractor rewards directly to cross-task downstream gains and by simultaneously refining a solver to exploit informative skills and ignore spurious ones [2605.10663].

Beyond RL replay, experience-based expensive optimization treats each expensive black-box problem as part of a family of related tasks. Shared task-independent parameters are meta-learned for a deep kernel learning surrogate and then adapted to a target task using only $1d$ solutions from the target task, with $d$ the dimension of the decision space. The learned “experience” becomes the initialization of the surrogate, enabling regression-based surrogate-assisted evolutionary algorithms to start from $d$ evaluations instead of the usual $10d$ or more [2304.04166].

A related but training-free line appears in Training-Free GRPO. Vanilla GRPO uses normalized numeric advantage
\[
\hat A_i = \frac{r_i-\mu_r}{\sigma_r},
\]
whereas Training-Free GRPO replaces it by a textual “semantic advantage” $A_{\text{text},i}$ distilled from grouped rollouts and accumulated into an experience library $\mathcal E$. At inference time, $\mathcal E$ is inserted into the prompt as a learned token prior:
\[
\pi_{\mathrm{TF}}(o\mid q')=\pi_\theta(o \mid [c;q']).
\]
No parameter update is performed. This is not replay prioritization in the strict Q-learning sense, but it preserves the central theme of relative experience evaluation: experience is ranked, distilled, and re-injected according to its group-relative usefulness [2510.08191].

## 6. Empirical findings, limitations, and recurring misconceptions

The original replay-priority formulation was implemented in “soft DQN” on nine Atari games. The reported baselines were uniform replay, PER with priority $\propto |\mathrm{TD}|$, and VER with priority $\propto \rho_{\max}|\mathrm{TD}|$. All agents trained for $40\,\mathrm M$ frames with the Mnih et al. network, $\beta=0.05$, $\alpha_{\mathrm{IS}}=0.6$, buffer $=1\,\mathrm M$, and batch $=32$. Uniform replay provided the baseline; PER yielded $+11.8\%$ average gain over uniform; VER yielded $+18.0\%$ gain over uniform and $+8.5\%$ gain over PER, and typically converged faster [2102.03261].

Other experience-optimization schemes report gains under different objectives. MAC-PO converges faster and attains $\approx 100\%$ capture rate on Predator-Prey under both no-punishment and heavy-punishment settings, yields $+10$–$16$ pp higher win-rates than uniform replay, PER, DisCor, and ReMERN on SMAC micromanagement, and improves super-hard MMM2 by $4$–$16$ pp over several MARL baselines; disabling the joint-action term drops final win-rate by $\sim 18$ pp [2302.10418]. ERO adds only $\sim 5$–$10\%$ extra CPU/GPU time and substantially accelerates learning on eight continuous-control benchmarks, while PER gives mixed or negligible gains for DDPG on these tasks [1906.08387]. VRER reports average-return improvements such as $+43\%$ for PPO-VRER on CartPole, $+23\%$ for TRPO-VRER on Hopper, and up to $+364\%$ for A2C-VRER on InvertedPendulum, alongside reported gradient-variance reductions of $20$–$30\%$ or $20\%$–$40\%$ depending on the experiment [2110.08902, 2602.05379].

In self-evolving language-agent settings, Evolving-RL reports on ALFWorld unseen tasks: GRPO $(w/\ \text{skills})$ at $44.6\%$ success, Evolving-RL $(w/\ \text{skills})$ at $88.6\%$ success, and Evolving-RL $(w/o\ \text{skills})$ at $81.1\%$ success; on Mind2Web overall action accuracy it reports $22.83\%$ for GRPO $(w/o\ \text{skills})$, $30.87\%$ for Evolving-RL $(w/\ \text{skills})$, and $28.05\%$ for Evolving-RL $(w/o\ \text{skills})$. The ALFWorld ablation further shows that solver-only improves unseen tasks to $70.3\%$, extractor-only reaches $73.7\%$ on seen tasks but only $27.6\%$ on unseen tasks, and full co-evolution reaches $98.6\%$ on seen tasks and $88.6\%$ on unseen tasks [2605.10663]. Training-Free GRPO reports AIME24/AIME25 gains from $68.6\%/52.9\%$ to $72.6\%/54.0\%$ without tools, gains from $80.0\%/67.9\%$ to $82.7\%/73.3\%$ with ReAct and a code interpreter, and a WebWalkerQA pass@1 increase from $63.2\%$ to $67.8\%$, with $\mathcal E$ distilled in $3$ epochs at cost $\approx \$18$ [2510.08191]. In expensive optimization, experience-based surrogate initialization uses only $d$ initial evaluations on the target task and still matches or beats standard MOEA/D-EGO with $90$ fewer initialization runs across DTLZ1–DTLZ6 [2304.04166].

Several misconceptions recur across this literature. One is that experience value is equivalent to TD error; the value-perspective results show an upper-bound relation, not an identity, and in soft Q-learning the bound is modulated by on-policyness, with no analogous uniform lower bound for $\mathrm{PIV}^{\text{soft}}$ [2102.03261]. A second is that more historical reuse is always beneficial; VRER’s theoretical contribution is precisely to formalize the bias-variance trade-off, where older samples can reduce variance while increasing bias [2110.08902, 2602.05379]. A third is that experience optimization is only a replay-buffer problem; Evolving-RL, experience-based surrogate adaptation, and Training-Free GRPO show that experience can also be extracted as skills, internalized into model parameters, or distilled into a token prior [2605.10663, 2304.04166, 2510.08191].

The cumulative picture is therefore heterogeneous but coherent. Relative Experience Optimization, in its narrow sense, is the use of $\rho_{\max}|\mathrm{TD}|$ as an upper-bound priority for replay. In a broader sense, it names a research direction in which the usefulness of experience is evaluated relative to current policy, downstream transfer benefit, coordination structure, or estimator variance. This suggests that the unifying question is not whether experience should be reused, but how its marginal utility should be formalized for the learning system at hand.

Source: https://www.emergentmind.com/topics/relative-experience-optimization