Rational Policy Gradient in Multi-Agent RL

• Rational Policy Gradient is a multi-agent RL method that enforces rationality constraints to avoid self-sabotage in adversarial training.
• It decomposes agent roles into a base policy and a manipulator, using higher-order gradients through unrolled updates to align training with realistic partner responses.
• Empirical results show that RPG maintains robust cross-play rewards and uncovers informative weaknesses in cooperative and general-sum game environments.

Searching arXiv for the cited papers to ground the article in the current literature. Rational Policy Gradient (RPG) is a multi-agent reinforcement-learning framework introduced as the algorithmic method for Rationality-preserving Policy Optimization (RPO), a formalism that augments adversarial optimization with a rationality constraint. In this usage, an agent is kept rational by requiring its policy to remain optimal with respect to some possible partner policy, which is intended to prevent the self-sabotage that arises when adversarial objectives are naively deployed in cooperative or general-sum games (Lauffer et al., 12 Nov 2025). The acronym is highly overloaded across the recent literature: nearby arXiv papers use RPG to denote Reparameterization Policy Gradient, Robust Policy Gradient, Regularized Policy Gradient, Reward Policy Gradient, and Reward-Randomized Policy Gradient (Zhong et al., 3 Feb 2026, Kumar et al., 2023, Zhang et al., 23 May 2025, Lan et al., 2021, Tang et al., 2021).

1. Terminological scope

In current arXiv usage, “RPG” is not a stable term. Strictly speaking, “Rational Policy Gradient” refers to the multi-agent formalism of “Robust and Diverse Multi-Agent Learning via Rational Policy Gradient” (Lauffer et al., 12 Nov 2025), while several adjacent literatures use the same acronym for unrelated methods.

Expansion of RPG	Representative paper	Context
Rational Policy Gradient	"Robust and Diverse Multi-Agent Learning via Rational Policy Gradient" (Lauffer et al., 12 Nov 2025)	Adversarial optimization in cooperative and general-sum multi-agent learning
Reparameterization Policy Gradient	"Reparameterization Flow Policy Optimization" (Zhong et al., 3 Feb 2026)	Model-based RL with differentiable dynamics
Robust Policy Gradient	"Policy Gradient for Rectangular Robust Markov Decision Processes" (Kumar et al., 2023)	Rectangular robust MDPs
Regularized Policy Gradient	"On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning" (Zhang et al., 23 May 2025)	KL-regularized off-policy RL for LLM reasoning
Reward Policy Gradient	"Model-free Policy Learning with Reward Gradients" (Lan et al., 2021)	Model-free RL with reward derivatives
Reward-Randomized Policy Gradient	"Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization" (Tang et al., 2021)	Diverse equilibria in multi-agent games

This ambiguity is not merely bibliographic. The multi-agent “rational” usage is anchored in a specific game-theoretic definition of rationality, whereas the other usages refer to robustness, regularization, reparameterization, or reward-based estimators. Any technical discussion therefore depends on which expansion is intended.

2. Game-theoretic setting and the self-sabotage problem

The Rational Policy Gradient paper works in a general-sum partially observable stochastic game. For a joint policy $(\pi_i,\pi_{-i})$, agent $i$ has expected discounted return

$$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$

and best-response correspondence

$$BR_i(\pi_{-i}) = \argmax_{\pi_i \in \Pi_i} U_i(\pi_i,\pi_{-i}).$$

The exposition is primarily two-player, but the formalism is stated for the general multi-agent case (Lauffer et al., 12 Nov 2025).

The motivating problem is adversarial optimization. In zero-sum settings, adversarial training is structurally aligned with the game, because minimizing the opponent’s reward is equivalent to maximizing one’s own reward. In cooperative and general-sum settings, the same objective can become irrational in the underlying game. The paper defines self-sabotage as the case where the optimization problem pushes an agent toward behavior that acts against its own incentive in the underlying game.

A canonical example is a cooperative matrix game in which the victim is the row player and the adversary is the column player. Under naive adversarial training,

$$\max_{\pi_{\text{victim}}} U_{\text{victim}}(\pi_{\text{victim}},\pi_{\text{adversary}}),\qquad \min_{\pi_{\text{adversary}}} U_{\text{victim}}(\pi_{\text{victim}},\pi_{\text{adversary}}),$$

the adversary chooses the action that trivially minimizes the victim’s payoff even though this also lowers its own payoff. The result is training collapse: the adversary blocks task completion rather than exposing informative weaknesses. The paper’s central claim is that this failure mode explains why adversarial optimization methods that are successful in zero-sum games can fail in cooperative settings (Lauffer et al., 12 Nov 2025).

3. Rationality-preserving Policy Optimization

RPO formalizes a constrained version of adversarial optimization. A policy $\pi_i$ is called rational if there exists some possible co-policy $\pi'_{-i}$ such that $\pi_i$ is a best response to it:

$$\pi_i \text{ is rational } \iff \exists \pi'_{-i} \in \Pi_{-i} \text{ such that } \pi_i \in BR_i(\pi'_{-i}).$$

This is weaker than Nash optimality, but it excludes policies that are never optimal for any partner and therefore excludes purely self-destructive behavior (Lauffer et al., 12 Nov 2025).

For an adversarial objective $O_i(\pi_1,\dots,\pi_m)$, RPO is

$i$0

The rationality constraint is vacuous for non-adversarial agents that already maximize their own reward, but it is substantive for agents whose training objective differs from their environment payoff. The paper emphasizes that, in zero-sum games, the RPO constraint does not change the solution set: the unconstrained adversarial optimum is already a best response to the current opponent.

In the cooperative matrix example, the distinction is concrete. The unconstrained adversary chooses the sabotaging action $i$1. Under RPO, that action is excluded because it is never a best response to any partner policy. The adversary instead chooses an action such as $i$2 that is rational with respect to some possible victim policy, thereby revealing a weakness without collapsing the task. The paper further states that, in this modified problem, the only equilibrium is for the victim to randomize uniformly between its two actions, yielding robust payoff $i$3 (Lauffer et al., 12 Nov 2025).

4. Rational Policy Gradient as the algorithmic realization of RPO

RPG implements RPO by splitting each adversarial role into a base agent and a manipulator. The base agent is the policy that will ultimately be used; it is trained only to maximize its own return, which preserves rationality. The manipulator does not act as the final policy. Instead, it shapes the training environment of the base agent so that the resulting base policy optimizes the adversarial objective indirectly (Lauffer et al., 12 Nov 2025).

For base agent $i$4, the training objective is

$i$5

where $i$6 denotes manipulator policies acting as the training partners. For the manipulator associated with adversarial objective $i$7, the objective is

$i$8

The key mechanism is opponent shaping: manipulators optimize the adversarial objective by differentiating through the learning updates of the base agents.

With parameters $i$9 for base policies and $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$0 for manipulators, the base update is a standard policy-gradient step,

$$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$1

while the manipulator update is a higher-order gradient through those base updates,

$$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$2

The paper generalizes this to an $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$3-step lookahead: base parameters are copied, unrolled for several learning steps against the manipulators, and the manipulator objective is differentiated through the full unroll. This makes RPG an opponent-shaping method in the style of higher-order meta-gradient RL rather than a conventional single-level policy gradient.

Two additional ingredients are central. First, the manipulator loss uses Loaded DiCE so that higher-order score-function gradients can be estimated through the unrolled updates. Second, the paper adds a small amount of partner-play data, weighted by a coefficient $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$4, because base agents are trained mostly against manipulators but are ultimately evaluated against other base agents. This regularization is intended to reduce distribution shift (Lauffer et al., 12 Nov 2025).

5. Algorithmic realization and empirical behavior

The implementation is actor-critic rather than PPO. The paper explicitly avoids PPO because clipping interferes with higher-order gradients. Policies are feed-forward neural networks; matrix games, STORM, and Overcooked use $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$5 MLPs, while Hanabi uses a $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$6 MLP. Each pair of agents has a separate centralized critic. Training uses on-policy trajectories, GAE$$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$7, entropy regularization, and gradient norm clipping for manipulators. The lookahead depth $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$8 is treated as a stabilizing factor; larger $$U_i(\pi_i,\pi_{-i}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t,a_t,s_{t+1})\right],$$9 can stabilize learning and reach unstable mixed equilibria in matrix games (Lauffer et al., 12 Nov 2025).

The empirical study evaluates several RPG variants. AD-RPG is the rationality-preserving version of adversarial diversity. AP-RPG is the rationality-preserving version of adversarial policy search. AT-RPG and PAIRED-RPG instantiate the same template for adversarial training and regret-based environment design. The environments include matrix games, STORM, Overcooked, and Hanabi.

Several recurring findings are reported. In STORM and in Overcooked’s Cramped Room, naive adversarial-diversity baselines and CoMeDi can produce sabotage: cross-play reward collapses, and in sabotage-STORM the policy learns to stand in the penalty square. AD-RPG instead maintains positive cross-play reward while still lowering cross-play relative to self-play, which the paper interprets as genuine diversity rather than self-sabotage. In Overcooked layouts such as Forced Coordination, Coordination Ring, and Counter Circuit, AD-RPG yields higher intra-population cross-play than self-play baselines and standard adversarial-diversity baselines. In Hanabi, AD-RPG also maintains good cross-play across seeds (Lauffer et al., 12 Nov 2025).

For adversarial-example generation, the paper studies unobserved STORM. Naive adversarial policy search always drives victim reward to $$BR_i(\pi_{-i}) = \argmax_{\pi_i \in \Pi_i} U_i(\pi_i,\pi_{-i}).$$0 by self-sabotage and therefore does not distinguish weak victims from strong ones. AP-RPG and PAIRED-A-RPG instead find non-trivial weaknesses: weaker victims lose substantial reward, whereas more robust victims retain high reward under attack. In Overcooked Cramped Room, AP-RPG is reported to discover a subtle incompatibility in path convention: the victim expects a clockwise coordination pattern, while the attacker induces a rational but incompatible counterclockwise pattern (Lauffer et al., 12 Nov 2025).

6. Broader interpretations and open issues

The term “rational” has also appeared in a looser methodological sense outside the 2025 multi-agent RPG formalism. “Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs” treats a rational policy-gradient design as one with principled step-size selection, no oracle access to hidden problem parameters, and convergence guarantees for softmax policy gradients in bandits and tabular MDPs (Lu et al., 2024). “Is the Policy Gradient a Gradient?” argues that a principled policy-gradient method should start from a precise objective and implement its gradient faithfully rather than rely on the common discountless update direction, which is shown not to be the gradient of any function in general (Nota et al., 2019). These usages are not algorithmically identical to Rational Policy Gradient in the multi-agent sense, but they indicate that “rational” also functions as a normative label for objective-faithful policy-gradient design.

Within the specific RPO/RPG framework, the paper is explicit about its current limitations. It does not provide a formal guarantee that RPG converges to globally rational strategies or to exact RPO solutions in general. Higher-order gradients and extra manipulator agents increase computational cost, and the method is substantially slower than simpler adversarial-diversity baselines. Sample efficiency is constrained by the variance of higher-order gradient estimators, even with Loaded DiCE and large batches. The authors therefore position formal convergence theory, more sample-efficient opponent shaping, and broader integration with other adversarial optimization methods as open directions rather than settled results (Lauffer et al., 12 Nov 2025).

Taken together, Rational Policy Gradient denotes a specific attempt to reconcile adversarial optimization with game-theoretic rationality in cooperative and general-sum multi-agent RL. Its distinctive contribution is not a new score-function estimator or a new trust-region rule, but a structural separation between rational base policies and opponent-shaping manipulators, designed to turn adversarial pressure into informative weaknesses rather than self-sabotaging collapse.