---
title: Dual-Perturbation Reward Optimization
url: https://www.emergentmind.com/topics/dual-perturbation-reward-optimization-algorithm
type: topic
---

# Dual-Perturbation Reward Optimization

Dual-perturbation reward optimization designates optimization schemes in which reward-linked perturbations are used to control learning dynamics or equilibrium selection through coupled signals. In the formulations considered here, the term covers two technically distinct constructions. In "Differentiable Arbitrating in Zero-sum Markov Games" [2302.10058], a designer perturbs the reward by $\delta$ and optimizes that perturbation through a bi-level problem whose lower level is an entropy-regularized Nash equilibrium. In "PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization" [2505.17714], policy entropy $H_t$ and reward progression $\Delta R_t$ are fused into a single bounded clipping threshold $\epsilon_t$, producing a phase-aware adaptive trust region for PPO. Both formulations treat perturbation not as an auxiliary heuristic but as an explicit optimization variable or control signal tied to reward-sensitive behavior.

## 1. Scope and conceptual variants

The two formulations differ in what is perturbed, what is optimized, and what object mediates the effect of the perturbation.

| Framework | Perturbed quantity | Immediate objective |
|---|---|---|
| Differentiable Arbitrating | Reward modified as $r+\delta$ | Induce a desirable Nash equilibrium |
| PPO-BR | PPO clipping threshold $\epsilon_t$ via entropy and reward signals | Balance exploration and convergence |

In the zero-sum Markov-game setting, the perturbation is literal reward modification: the designer chooses $\delta$ to minimize a system loss evaluated at the Nash equilibrium of the perturbed game. In PPO-BR, by contrast, the reward signal enters through the smoothed return delta $\Delta R_t=R_t-R_{t-k}$, which contracts the trust region when rewards plateau, while policy entropy expands it under high uncertainty [2302.10058][2505.17714].

A common source of confusion is to treat these as instances of the same algorithm. They are not. The first is a bi-level reward-perturbation method for zero-sum Markov games; the second is a drop-in replacement for PPO’s clipping logic. The shared structural feature is dual control via coupled signals: in one case, upper-level reward perturbation and lower-level equilibrium computation; in the other, entropy-driven expansion and reward-guided contraction.

## 2. Bi-level reward perturbation in zero-sum Markov games

In "Differentiable Arbitrating in Zero-sum Markov Games" [2302.10058], the designer solves an upper-level optimization over the perturbation $\delta$:
$$
\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.
$$
Here $F$ is the system loss, $\pi^*(r+\delta)$ is the equilibrium induced by the perturbed reward, and $\tfrac{\lambda}{2}\|\delta\|^2$ is a regularizer on the perturbation.

The lower level solves for an entropy-regularized Nash equilibrium under the modified reward $r+\delta$. Writing $\pi=(\pi^1,\pi^2)$, the formulation is
$$
\pi^*(r+\delta)
\;=\;
\argmin_{\pi^2}\;\max_{\pi^1}
\;V^{(1)}_{\pi}(s;\,r+\delta)\;-\;V^{(2)}_{\pi}(s;\,r+\delta)
\;+\;\eta\;\bigl[H(\pi^1)-H(\pi^2)\bigr].
$$
The entropy penalty has strength $\eta\ge 0$ and is introduced to ensure uniqueness. Equivalently, defining the regularized payoff
$\mathcal{L}(\pi^1,\pi^2;\,r+\delta)=V^{(1)}_\pi-V^{(2)}_\pi$,
the lower level is written as
$$
\pi^*(r+\delta)\;=\;
\argmin_{\pi^2}\;\max_{\pi^1}\;
\mathcal{L}(\pi^1,\pi^2;\,r+\delta)\;+\;\eta\bigl[D(\pi^1)-D(\pi^2)\bigr].
$$

This construction makes arbitrating a bi-level optimization problem. The upper level expresses the designer’s preference over equilibria, while the lower level enforces game-theoretic consistency through the regularized Nash solution. A plausible implication is that the perturbation is not merely reward shaping in the usual ad hoc sense; it is selected so that the equilibrium of the modified game optimizes a higher-level criterion.

## 3. Differentiation through equilibrium and convergence structure

Because the equilibrium $\pi^*$ depends implicitly on $\delta$, the upper-level gradient must include the equilibrium sensitivity term. Using the low-level parameterization $\phi$, the total gradient is
$$
\nabla_\delta\,F\bigl(\delta,\phi^*(\delta)\bigr)
\;=\;
\nabla_\delta F\;+\;
\bigl[\tfrac{\partial \phi^*}{\partial\delta}\bigr]^\top
\,\nabla_\phi F
\quad\text{at }\phi=\phi^*(\delta).
$$
To compute $\partial\phi^*/\partial\delta$, the method differentiates the first-order equilibrium condition. Defining
$$
u_\delta(\phi)\;=\;
\begin{pmatrix}
\nabla_{\phi^1}V^{(1)}_{\pi_\phi}(r+\delta)\\[3pt]
\nabla_{\phi^2}V^{(2)}_{\pi_\phi}(r+\delta)
\end{pmatrix},
\quad\text{and}\quad
u_\delta\bigl(\phi^*(\delta)\bigr)=0,
$$
differentiation with respect to $\delta$ gives
$$
\nabla_\delta u_\delta \;+\;
\bigl[\nabla_\phi u_\delta\bigr]\;\tfrac{\partial\phi^*}{\partial\delta}
\;=\;0
\;\;\Longrightarrow\;\;
\tfrac{\partial\phi^*}{\partial\delta}
\;=\;
-\,\bigl[\nabla_\phi u_\delta\bigr]^{-1}
\,\nabla_\delta u_\delta.
$$
Substitution yields the explicit upper-level gradient
$$
\boxed{
\nabla_\delta F_*(\delta)
=
\nabla_\delta F
\;-\;
\bigl(\nabla_\delta u_\delta\bigr)^\top
\bigl[\nabla_\phi u_\delta\bigr]^{-1}
\,\nabla_\phi F
}.
$$

The implementation is high-level but explicit. A black-box NE solver $\Xi$ finds the regularized equilibrium for $r+\delta_k$, the gradient
$$
g \leftarrow \nabla_\delta F(\delta_k,\phi_k)
- \nabla_\delta u(\phi_k)^T [\nabla_\phi u(\phi_k)]^{-1} \nabla_\phi F(\delta_k,\phi_k)
$$
is computed, and the perturbation is updated by
$$
\delta_{k+1}\leftarrow \delta_k-\beta\cdot g.
$$
The paper states that step 1 can use any convergent NE routine, including PEM or entropic OMWU, and that the required matrices are Hessians available by back-propagating through rollouts under $\pi_k$ [2302.10058].

The convergence discussion has two layers. For the lower level, Policy-Extragradient Method or Entropy-OMWU attain an $\epsilon$-approximate regularized NE in
$\widetilde O\bigl((1-\gamma)^{-2}\lambda^{-1}\ln^2(1/\epsilon)\bigr)$
or
$\widetilde O\bigl((1-\gamma)^{-1}\lambda^{-1}\ln(1/\epsilon)\bigr)$
iterations, respectively. For the upper level, under $L$-smoothness of $F^*$, gradient descent on $\delta$ satisfies
$$
\min_{k=0,\dots,T}\bigl\|\nabla F_*(\delta_k)\bigr\|^2
\;\le\;
\frac{2L\bigl(F_*(\delta_0)-F_*(\delta^*)\bigr)}{T+1}
\;=\;O(1/T).
$$
The stated assumptions are that $F^*(\delta)=F(\delta,\phi^*(\delta))$ is bounded and $L$-smooth, and that the lower-level NE mapping $\phi^*(\delta)$ is well-behaved, strongly monotone under regularization, so $\nabla_\phi u$ is invertible with a uniform condition number.

The role of dual variables is interpretive but explicit in the summary: the equilibrium conditions $u_\delta(\phi)=0$ can be viewed as the KKT system of the inner min-max, and $\nabla_\phi u$ is the Jacobian of that system, or “Hessian of the inner Lagrangian.” This suggests a direct application of the implicit-function theorem to a variational-inequality constraint.

## 4. Dual-signal entropy-reward adaptation in PPO-BR

"PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization" [2505.17714] addresses a different problem: standard PPO uses a fixed clipping threshold $\epsilon_0$ in the surrogate
$$
L_{\text{clip}}(\theta)=E_t[\min(r_t(\theta)\hat A_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon_0,1+\epsilon_0)\hat A_t)].
$$
The stated failure mode is phase dependence. Early in training, $\epsilon_0$ may be too small, “starving” exploration because high-entropy policies are over-clipped; late in training, $\epsilon_0$ may be too large, allowing destabilizing updates near convergence.

PPO-BR introduces two complementary signals. Policy entropy is
$$
H_t = E_{a\sim\pi_\theta(\cdot|s_t)}[-\log \pi_\theta(a|s_t)],
$$
and reward progression is
$$
\Delta R_t = R_t - R_{t-k},
$$
a smoothed return delta over $k$ episodes. These are normalized by $\phi(H_t)\in[0,1]$ and $\psi(\Delta R_t)\in[0,1]$, with hyperparameters $\epsilon_0>0$, $\lambda_1\ge 0$, and $\lambda_2\ge 0$.

The entropy-driven expansion term is
$$
\epsilon_{\text{up}}(t)=\epsilon_0\cdot[1+\lambda_1\cdot\tanh(\phi(H_t))].
$$
When $H_t$ is large, $\phi(H_t)\to 1$, $\tanh\to\sim 1$, and $\epsilon_{\text{up}}\to\epsilon_0(1+\lambda_1)$.

The reward-guided contraction term is
$$
\epsilon_{\text{down}}(t)=\epsilon_0\cdot[1-\lambda_2\cdot\tanh(\psi(\Delta R_t))].
$$
When reward improvement $\Delta R_t$ is small, $\psi(\Delta R_t)\to 0$, $\tanh\to 0$, and $\epsilon_{\text{down}}\to\epsilon_0$. When $\Delta R_t$ saturates so that $\psi\to 1$, $\tanh\to\sim 1$, and $\epsilon_{\text{down}}\to\epsilon_0(1-\lambda_2)$.

Rather than applying two separate bounds, PPO-BR fuses the signals additively:
$$
\epsilon_t = \epsilon_0 \cdot [1 + \lambda_1 \tanh(\phi(H_t)) - \lambda_2 \tanh(\psi(\Delta R_t))].
$$
Since $\tanh\in[0,1]$,
$$
\epsilon_{\min}=\epsilon_0(1-\lambda_2)\le \epsilon_t \le \epsilon_0(1+\lambda_1)=\epsilon_{\max},
$$
so the adaptive trust region remains bounded at every update, which is stated as Lemma 1. Substituting $\epsilon_t$ into PPO yields
$$
L_{\text{PPO-BR}}(\theta)
=
E_t \left[
\min(r_t(\theta)\hat A_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon_t,1+\epsilon_t)\hat A_t)
\right].
$$

This formulation does not perturb the environment reward directly. It perturbs the trust-region width using reward progression and entropy as phase-aware signals. That distinction is central when comparing PPO-BR to reward-optimization methods in games.

## 5. Algorithmic workflow and theoretical guarantees of PPO-BR

The PPO-BR update is presented as a drop-in replacement for PPO’s clipping logic. Given policy $\pi_\theta$, value $V_\phi$, base clip $\epsilon_0$, hyperparameters $\lambda_1,\lambda_2$, and reward window $k$, each iteration performs rollout collection, computes advantage estimates $\hat A_t$ such as via GAE, computes policy entropy $H_t$, computes smoothed return deltas $\Delta R_t=R_t-R_{t-k}$, normalizes them as $e_H=\phi(H_t)$ and $e_R=\psi(\Delta R_t)$, then forms
$$
\epsilon_t=\epsilon_0[1+\lambda_1\tanh(e_H)-\lambda_2\tanh(e_R)]
$$
and clamps it to
$$
\epsilon_t \leftarrow \min(\max(\epsilon_t,\epsilon_0(1-\lambda_2)),\epsilon_0(1+\lambda_1)).
$$
The surrogate loss is then evaluated with clipping at $[1-\epsilon_t,1+\epsilon_t]$, and parameters are updated by
$$
\theta \leftarrow \theta + \alpha\nabla_\theta L_{\text{clip}},
\qquad
\phi \leftarrow \phi - \beta\nabla_\phi\|V_\phi - R_t\|^2.
$$

The theoretical statements are concise. Lemma 1 gives bounded adaptation: if $\phi(H_t),\psi(\Delta R_t)\in[0,1]$ and $\lambda_1,\lambda_2\ge 0$, then $\epsilon_t\in[\epsilon_0(1-\lambda_2),\epsilon_0(1+\lambda_1)]$. Theorem 1 gives monotonic improvement: under the usual PPO assumptions, namely unbiased advantage estimates $\hat A_t$ and bounded $\epsilon_t$, the expected return does not decrease after each policy update,
$$
E[J(\theta_{k+1})] \ge E[J(\theta_k)].
$$
The accompanying assumptions are that the advantage estimator is approximately unbiased, the normalization functions and $\tanh$ ensure differentiability and boundedness of $\epsilon_t$, and no auxiliary networks beyond standard PPO are introduced [2505.17714].

A plausible implication is that the method aims to preserve PPO’s stability profile while replacing the static trust region with a bounded adaptive one. The paper states this as an attempt to preserve PPO’s monotonic improvement guarantees while encouraging bold updates when exploration is needed and conservative updates when stability is needed.

## 6. Empirical profile, comparisons, and interpretive cautions

The reported PPO-BR experiments use 6 OpenAI Gym / MuJoCo tasks: CartPole, LunarLander, Hopper, HalfCheetah, Walker2D, and Humanoid. Baselines are PPO, KL-PPO, entropy-only PPO, reward-only PPO, and Annealed PPO. The summary reports, relative to standard PPO and with all $p<0.001$ under a Wilcoxon test, 29.1% faster convergence as the env-averaged reduction in steps to reach target return, up to 31.3% higher final returns with Humanoid reported as 1600→2100, 2.3× lower reward variance in high-dimensional tasks, 44–52% variance reduction in continuous-control benchmarks, and less than 1.8% runtime overhead because only scalar clip updates are added [2505.17714].

The ablation results are presented in phase-specific terms. The entropy-only variant improves early returns but later oscillates. The reward-only variant is stable but slower to explore. Full PPO-BR is reported to unify both benefits and to dominate in both phases. The abstract separately states that the mechanism achieves the gains with only five lines of code change and that it outperforms five SOTA baselines with less than 2% overhead.

The comparison set also clarifies scope. Standard PPO uses a single fixed $\epsilon_0$ and therefore has a brittle exploration-convergence trade-off with no phase awareness. GRPO, described as Group Relative Policy Optimization, is characterized as critic-free ranking for LLMs, with no entropy control, no bounded trust region, and limitation to preference-based LLM fine-tuning. PPO-BR is described as a unified entropy-reward mechanism applicable to both language models and general reinforcement learning environments, including discrete/continuous and low/high-dimensional domains, and as extending to LLM/RLHF [2505.17714].

For the zero-sum Markov-game formulation, a corresponding caution is that differentiable arbitrating is not presented as a heuristic that directly rewrites both players’ objectives. Its central mechanism is a reward perturbation $\delta$, a regularized NE at the lower level, and end-to-end differentiation through that equilibrium via implicit-function or dual-sensitivity calculations. The abstract further emphasizes that the method only requires a black-box solver for the regularized Nash equilibrium and develops convergence analysis for the proposed framework with proper black-box NE solvers [2302.10058].

Source: https://www.emergentmind.com/topics/dual-perturbation-reward-optimization-algorithm