Dual-Perturbation Reward Optimization

• Dual-perturbation reward optimization is defined by two formulations: one using bi-level reward perturbation for zero-sum Nash equilibria and the other adapting PPO clipping via entropy and reward signals.
• The first formulation employs a bi-level optimization that perturbs the reward with δ and leverages implicit differentiation to guide equilibrium selection.
• The PPO-BR method fuses normalized policy entropy and reward progression to dynamically adjust trust regions, achieving faster convergence and improved stability.

Dual-perturbation reward optimization designates optimization schemes in which reward-linked perturbations are used to control learning dynamics or equilibrium selection through coupled signals. In the formulations considered here, the term covers two technically distinct constructions. In "Differentiable Arbitrating in Zero-sum Markov Games" (Wang et al., 2023), a designer perturbs the reward by $\delta$ and optimizes that perturbation through a bi-level problem whose lower level is an entropy-regularized Nash equilibrium. In "PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization" (Rahman, 23 May 2025), policy entropy $H_t$ and reward progression $\Delta R_t$ are fused into a single bounded clipping threshold $\epsilon_t$, producing a phase-aware adaptive trust region for PPO. Both formulations treat perturbation not as an auxiliary heuristic but as an explicit optimization variable or control signal tied to reward-sensitive behavior.

1. Scope and conceptual variants

The two formulations differ in what is perturbed, what is optimized, and what object mediates the effect of the perturbation.

Framework	Perturbed quantity	Immediate objective
Differentiable Arbitrating	Reward modified as $r+\delta$	Induce a desirable Nash equilibrium
PPO-BR	PPO clipping threshold $\epsilon_t$ via entropy and reward signals	Balance exploration and convergence

In the zero-sum Markov-game setting, the perturbation is literal reward modification: the designer chooses $\delta$ to minimize a system loss evaluated at the Nash equilibrium of the perturbed game. In PPO-BR, by contrast, the reward signal enters through the smoothed return delta $\Delta R_t=R_t-R_{t-k}$, which contracts the trust region when rewards plateau, while policy entropy expands it under high uncertainty (Wang et al., 2023, Rahman, 23 May 2025).

A common source of confusion is to treat these as instances of the same algorithm. They are not. The first is a bi-level reward-perturbation method for zero-sum Markov games; the second is a drop-in replacement for PPO’s clipping logic. The shared structural feature is dual control via coupled signals: in one case, upper-level reward perturbation and lower-level equilibrium computation; in the other, entropy-driven expansion and reward-guided contraction.

2. Bi-level reward perturbation in zero-sum Markov games

In "Differentiable Arbitrating in Zero-sum Markov Games" (Wang et al., 2023), the designer solves an upper-level optimization over the perturbation $\delta$:

$$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$

Here $H_t$0 is the system loss, $H_t$1 is the equilibrium induced by the perturbed reward, and $H_t$2 is a regularizer on the perturbation.

The lower level solves for an entropy-regularized Nash equilibrium under the modified reward $H_t$3. Writing $H_t$4, the formulation is

$H_t$5

The entropy penalty has strength $H_t$6 and is introduced to ensure uniqueness. Equivalently, defining the regularized payoff $H_t$7, the lower level is written as

$H_t$8

This construction makes arbitrating a bi-level optimization problem. The upper level expresses the designer’s preference over equilibria, while the lower level enforces game-theoretic consistency through the regularized Nash solution. A plausible implication is that the perturbation is not merely reward shaping in the usual ad hoc sense; it is selected so that the equilibrium of the modified game optimizes a higher-level criterion.

3. Differentiation through equilibrium and convergence structure

Because the equilibrium $H_t$9 depends implicitly on $\Delta R_t$0, the upper-level gradient must include the equilibrium sensitivity term. Using the low-level parameterization $\Delta R_t$1, the total gradient is

$\Delta R_t$2

To compute $\Delta R_t$3, the method differentiates the first-order equilibrium condition. Defining

$\Delta R_t$4

differentiation with respect to $\Delta R_t$5 gives

$\Delta R_t$6

Substitution yields the explicit upper-level gradient

$\Delta R_t$7

The implementation is high-level but explicit. A black-box NE solver $\Delta R_t$8 finds the regularized equilibrium for $\Delta R_t$9, the gradient

$\epsilon_t$0

is computed, and the perturbation is updated by

$\epsilon_t$1

The paper states that step 1 can use any convergent NE routine, including PEM or entropic OMWU, and that the required matrices are Hessians available by back-propagating through rollouts under $\epsilon_t$2 (Wang et al., 2023).

The convergence discussion has two layers. For the lower level, Policy-Extragradient Method or Entropy-OMWU attain an $\epsilon_t$3-approximate regularized NE in

$\epsilon_t$4

or

$\epsilon_t$5

iterations, respectively. For the upper level, under $\epsilon_t$6-smoothness of $\epsilon_t$7, gradient descent on $\epsilon_t$8 satisfies

$\epsilon_t$9

The stated assumptions are that $r+\delta$0 is bounded and $r+\delta$1-smooth, and that the lower-level NE mapping $r+\delta$2 is well-behaved, strongly monotone under regularization, so $r+\delta$3 is invertible with a uniform condition number.

The role of dual variables is interpretive but explicit in the summary: the equilibrium conditions $r+\delta$4 can be viewed as the KKT system of the inner min-max, and $r+\delta$5 is the Jacobian of that system, or “Hessian of the inner Lagrangian.” This suggests a direct application of the implicit-function theorem to a variational-inequality constraint.

4. Dual-signal entropy-reward adaptation in PPO-BR

"PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization" (Rahman, 23 May 2025) addresses a different problem: standard PPO uses a fixed clipping threshold $r+\delta$6 in the surrogate

$r+\delta$7

The stated failure mode is phase dependence. Early in training, $r+\delta$8 may be too small, “starving” exploration because high-entropy policies are over-clipped; late in training, $r+\delta$9 may be too large, allowing destabilizing updates near convergence.

PPO-BR introduces two complementary signals. Policy entropy is

$\epsilon_t$0

and reward progression is

$\epsilon_t$1

a smoothed return delta over $\epsilon_t$2 episodes. These are normalized by $\epsilon_t$3 and $\epsilon_t$4, with hyperparameters $\epsilon_t$5, $\epsilon_t$6, and $\epsilon_t$7.

The entropy-driven expansion term is

$\epsilon_t$8

When $\epsilon_t$9 is large, $\delta$0, $\delta$1, and $\delta$2.

The reward-guided contraction term is

$\delta$3

When reward improvement $\delta$4 is small, $\delta$5, $\delta$6, and $\delta$7. When $\delta$8 saturates so that $\delta$9, $\Delta R_t=R_t-R_{t-k}$0, and $\Delta R_t=R_t-R_{t-k}$1.

Rather than applying two separate bounds, PPO-BR fuses the signals additively:

$\Delta R_t=R_t-R_{t-k}$2

Since $\Delta R_t=R_t-R_{t-k}$3,

$\Delta R_t=R_t-R_{t-k}$4

so the adaptive trust region remains bounded at every update, which is stated as Lemma 1. Substituting $\Delta R_t=R_t-R_{t-k}$5 into PPO yields

$\Delta R_t=R_t-R_{t-k}$6

This formulation does not perturb the environment reward directly. It perturbs the trust-region width using reward progression and entropy as phase-aware signals. That distinction is central when comparing PPO-BR to reward-optimization methods in games.

5. Algorithmic workflow and theoretical guarantees of PPO-BR

The PPO-BR update is presented as a drop-in replacement for PPO’s clipping logic. Given policy $\Delta R_t=R_t-R_{t-k}$7, value $\Delta R_t=R_t-R_{t-k}$8, base clip $\Delta R_t=R_t-R_{t-k}$9, hyperparameters $\delta$0, and reward window $\delta$1, each iteration performs rollout collection, computes advantage estimates $\delta$2 such as via GAE, computes policy entropy $\delta$3, computes smoothed return deltas $\delta$4, normalizes them as $\delta$5 and $\delta$6, then forms

$\delta$7

and clamps it to

$\delta$8

The surrogate loss is then evaluated with clipping at $\delta$9, and parameters are updated by

$$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$0

The theoretical statements are concise. Lemma 1 gives bounded adaptation: if $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$1 and $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$2, then $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$3. Theorem 1 gives monotonic improvement: under the usual PPO assumptions, namely unbiased advantage estimates $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$4 and bounded $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$5, the expected return does not decrease after each policy update,

$$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$6

The accompanying assumptions are that the advantage estimator is approximately unbiased, the normalization functions and $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$7 ensure differentiability and boundedness of $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$8, and no auxiliary networks beyond standard PPO are introduced (Rahman, 23 May 2025).

A plausible implication is that the method aims to preserve PPO’s stability profile while replacing the static trust region with a bounded adaptive one. The paper states this as an attempt to preserve PPO’s monotonic improvement guarantees while encouraging bold updates when exploration is needed and conservative updates when stability is needed.

6. Empirical profile, comparisons, and interpretive cautions

The reported PPO-BR experiments use 6 OpenAI Gym / MuJoCo tasks: CartPole, LunarLander, Hopper, HalfCheetah, Walker2D, and Humanoid. Baselines are PPO, KL-PPO, entropy-only PPO, reward-only PPO, and Annealed PPO. The summary reports, relative to standard PPO and with all $$\delta^* \;=\;\argmin_{\delta\in\Delta}\;\Bigl\{\,F\bigl(\delta,\;\pi^*(\,r+\delta\,)\bigr)\;+\;\tfrac{\lambda}{2}\|\delta\|^2\Bigr\}\;.$$9 under a Wilcoxon test, 29.1% faster convergence as the env-averaged reduction in steps to reach target return, up to 31.3% higher final returns with Humanoid reported as 1600→2100, 2.3× lower reward variance in high-dimensional tasks, 44–52% variance reduction in continuous-control benchmarks, and less than 1.8% runtime overhead because only scalar clip updates are added (Rahman, 23 May 2025).

The ablation results are presented in phase-specific terms. The entropy-only variant improves early returns but later oscillates. The reward-only variant is stable but slower to explore. Full PPO-BR is reported to unify both benefits and to dominate in both phases. The abstract separately states that the mechanism achieves the gains with only five lines of code change and that it outperforms five SOTA baselines with less than 2% overhead.

The comparison set also clarifies scope. Standard PPO uses a single fixed $H_t$00 and therefore has a brittle exploration-convergence trade-off with no phase awareness. GRPO, described as Group Relative Policy Optimization, is characterized as critic-free ranking for LLMs, with no entropy control, no bounded trust region, and limitation to preference-based LLM fine-tuning. PPO-BR is described as a unified entropy-reward mechanism applicable to both LLMs and general reinforcement learning environments, including discrete/continuous and low/high-dimensional domains, and as extending to LLM/RLHF (Rahman, 23 May 2025).

For the zero-sum Markov-game formulation, a corresponding caution is that differentiable arbitrating is not presented as a heuristic that directly rewrites both players’ objectives. Its central mechanism is a reward perturbation $H_t$01, a regularized NE at the lower level, and end-to-end differentiation through that equilibrium via implicit-function or dual-sensitivity calculations. The abstract further emphasizes that the method only requires a black-box solver for the regularized Nash equilibrium and develops convergence analysis for the proposed framework with proper black-box NE solvers (Wang et al., 2023).