Near-Potential Policy Optimization (NePPO)

• NePPO is a framework that uses a learned player-independent potential to transform a general-sum game into a cooperative one, aligning incentive structures.
• It minimizes local discrepancies in unilateral reward differences using a zeroth-order optimization method combined with Monte Carlo rollouts for stable convergence.
• Empirical results demonstrate that NePPO reduces per-agent regret compared to MAPPO and IPPO in mixed cooperative–competitive environments.

Near-Potential Policy Optimization (NePPO) is a multi-agent reinforcement learning pipeline for general-sum Markov games that learns a player-independent potential function whose cooperative equilibrium approximates a Nash equilibrium of the original game. In its primary formulation, NePPO targets mixed cooperative–competitive environments under partial observability and continuous actions by minimizing a discrepancy between unilateral incentive differences in the original game and those induced by a learned potential; the resulting policy is returned as an approximate Nash equilibrium. A distinct later usage of the same name appears in work on reinforcement learning with verifiable rewards, where “Near-Potential Policy Optimization” is presented as a semantic reinterpretation of Near-Future Policy Optimization rather than the original MARL method, so the term denotes two related but technically different constructions in the 2026 literature (Kalanther et al., 7 Mar 2026, Qin et al., 22 Apr 2026).

1. Formal setting in general-sum Markov games

NePPO is formulated for an infinite-horizon discounted Partially-Observable Markov Game (POMG)

$$\mathcal{G} = \langle \mathcal{N}, \mathcal{S}, (\mathcal{A}_i)_{i\in\mathcal{N}}, \mathcal{P}, (\mathcal{Z}_i)_{i\in\mathcal{N}}, \mathcal{O}, (r_i)_{i\in\mathcal{N}}, \gamma, \rho\rangle,$$

where $\mathcal{N}=\{1,\dots,N\}$ is the agent set, $\mathcal{S}\subset\mathbb{R}^{d_s}$ is the continuous state space, $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$ is agent $i$’s continuous action space, $\mathcal{P}$ is the stochastic transition kernel, $\mathcal{Z}_i$ is the observation space of agent $i$, $\mathcal{O}_i$ is the observation kernel, $r_i$ is the stage reward, $\mathcal{N}=\{1,\dots,N\}$0 is the discount factor, and $\mathcal{N}=\{1,\dots,N\}$1 is the initial-state distribution. Each agent acts from its own history $\mathcal{N}=\{1,\dots,N\}$2 using a stochastic history-dependent policy $\mathcal{N}=\{1,\dots,N\}$3, with joint policy space $\mathcal{N}=\{1,\dots,N\}$4.

The discounted return of agent $\mathcal{N}=\{1,\dots,N\}$5 under joint policy $\mathcal{N}=\{1,\dots,N\}$6 is

$\mathcal{N}=\{1,\dots,N\}$7

A joint policy $\mathcal{N}=\{1,\dots,N\}$8 is an $\mathcal{N}=\{1,\dots,N\}$9-approximate Nash equilibrium if, for all agents $\mathcal{S}\subset\mathbb{R}^{d_s}$0 and all unilateral deviations $\mathcal{S}\subset\mathbb{R}^{d_s}$1,

$\mathcal{S}\subset\mathbb{R}^{d_s}$2

The paper’s starting point is that, in general-sum, partially observable, continuous-action environments, learning dynamics are unstable, convergence guarantees are uncommon outside special cases such as two-player zero-sum or fully cooperative games, and equilibria may be non-unique and qualitatively diverse. NePPO is designed specifically for this regime rather than for pure social-welfare optimization or independent return maximization (Kalanther et al., 7 Mar 2026).

2. Near-potential functions and the induced cooperative game

The conceptual core of NePPO is the replacement of the original general-sum game by a cooperative game induced by a player-independent potential. In classical normal-form potential games, an exact potential $\mathcal{S}\subset\mathbb{R}^{d_s}$3 satisfies

$\mathcal{S}\subset\mathbb{R}^{d_s}$4

for any agent $\mathcal{S}\subset\mathbb{R}^{d_s}$5, unilateral action change from $\mathcal{S}\subset\mathbb{R}^{d_s}$6 to $\mathcal{S}\subset\mathbb{R}^{d_s}$7, and fixed opponents’ actions $\mathcal{S}\subset\mathbb{R}^{d_s}$8. NePPO lifts this idea to Markov settings by working directly in policy space.

Its central object is the Markov near-potential function (MNPF). A function $\mathcal{S}\subset\mathbb{R}^{d_s}$9 is an MNPF with parameter $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$0 if

$\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$1

When $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$2, the game is a Markov potential game. NePPO does not require exact potentiality; instead, it exploits the weaker condition that a player-independent potential can approximate unilateral incentive differences closely enough near the equilibrium-relevant region of policy space.

Given such a $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$3, one defines a cooperative game $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$4 with the same dynamics as the original environment but with common utility $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$5 for all agents. If $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$6 is a Nash equilibrium of this cooperative game and $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$7 is an MNPF of $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$8 with parameter $\mathcal{A}_i\subset\mathbb{R}^{d_{a_i}}$9, then $i$0 is an $i$1-approximate Nash equilibrium of the original game. This proposition motivates the entire pipeline: rather than solving the original general-sum game directly, NePPO learns a potential whose cooperative equilibrium is strategically faithful to the original incentives (Kalanther et al., 7 Mar 2026).

3. Equilibrium-local discrepancy objective and guarantee

NePPO does not enforce the near-potential condition uniformly over all policies. Instead, it minimizes a local discrepancy metric centered on the cooperative equilibrium induced by the candidate potential. For each agent $i$2,

$i$3

where $i$4 is a Nash equilibrium of the cooperative game with common utility $i$5, and $i$6 is agent $i$7’s best response against the others fixed at the cooperative equilibrium. By construction, $i$8.

The main theorem states that if

$i$9

then $\mathcal{P}$0 is an $\mathcal{P}$1-Nash equilibrium of the original game. The proof uses the fact that $\mathcal{P}$2 maximizes $\mathcal{P}$3, yielding

$\mathcal{P}$4

Hence no player can gain more than $\mathcal{P}$5 by unilateral deviation.

This equilibrium-local criterion is strictly weaker than global potentiality. If the game is a potential game with potential $\mathcal{P}$6, then $\mathcal{P}$7 for all $\mathcal{P}$8. Conversely, the paper notes that there exist non-potential games with $\mathcal{P}$9, so vanishing local discrepancy is necessary for the targeted equilibrium guarantee but not sufficient to establish that the whole game is globally potential. This suggests that NePPO is best interpreted as an equilibrium-focused approximation framework rather than a general characterization of game structure (Kalanther et al., 7 Mar 2026).

4. Optimization problem, smoothing, and zeroth-order outer loop

To optimize the worst-agent discrepancy, NePPO parameterizes a one-stage function $\mathcal{Z}_i$0 with parameters $\mathcal{Z}_i$1 and defines

$\mathcal{Z}_i$2

The original objective is

$\mathcal{Z}_i$3

which NePPO approximates with the log-sum-exp smooth surrogate

$\mathcal{Z}_i$4

satisfying

$\mathcal{Z}_i$5

The optimization problem becomes

$\mathcal{Z}_i$6

The paper emphasizes that the method is model-free and does not require differentiability of either $\mathcal{Z}_i$7 or $\mathcal{Z}_i$8. Instead, it uses zeroth-order optimization with a two-point estimator. Sampling a random direction $\mathcal{Z}_i$9 uniformly from the unit sphere in $i$0, setting $i$1 and $i$2, NePPO computes

$i$3

and updates

$i$4

For each probe point, the inner loop has four stages: solve the cooperative game defined by $i$5 to obtain $i$6; compute each agent’s best response $i$7 against $i$8; estimate $i$9, $\mathcal{O}_i$0, $\mathcal{O}_i$1, and $\mathcal{O}_i$2 by Monte Carlo rollouts; and assemble

$\mathcal{O}_i$3

Cooperative optimization is implemented with HAPPO or MAPPO using $\mathcal{O}_i$4 as the one-stage reward, while best responses are computed with PPO with opponents fixed. The paper characterizes this as NePPO’s principal deviation from PPO or MAPPO: the learned potential enters only through the outer-loop objective, whereas the inner solvers remain standard RL components (Kalanther et al., 7 Mar 2026).

5. Empirical results and comparative position

The empirical study includes both a toy matrix game and a mixed cooperative–competitive partially observable environment. In the two-player two-action normal-form example, the potential is parameterized as $\mathcal{O}_i$5. The paper shows analytically that

$\mathcal{O}_i$6

and that the cooperative maximizer $\mathcal{O}_i$7 coincides with the true Nash equilibrium $\mathcal{O}_i$8. The zeroth-order outer loop converges to $\mathcal{O}_i$9, which lies in the minimizer interval and recovers the correct equilibrium. By contrast, MAPPO, when optimizing $r_i$0, converges to $r_i$1, which is not a Nash equilibrium in that game.

The main benchmark is the Multi-Particle Environment Simple World Comm, a mixed cooperative–competitive POMG with heroes and adversaries under partial observability and communication. In this setting, the one-stage potential is parameterized as

$r_i$2

where $r_i$3, $r_i$4, $r_i$5, and $r_i$6 is the softmax. Equilibrium quality is measured by maximum regret per agent, computed by training a PPO best response against frozen policies. The reported maximum regret values are 51.78 for MAPPO, 23.90 for IPPO, and 17.26 for NePPO, while MADDPG did not converge. The qualitative interpretation given in the paper is that MAPPO tends to optimize one team’s sum of rewards at the expense of the other, IPPO struggles with complex coordination, and NePPO reduces exploitability by learning a player-independent aggregation of rewards that better aligns with equilibrium incentives.

In comparative positioning, the paper distinguishes NePPO from independent learning, centralized-critic PPO variants, opponent modeling, differentiable games, PSRO-style meta-solvers, and potential-based shaping. Its stated novelty is the direct optimization of a player-independent potential whose cooperative equilibrium approximates a Nash equilibrium through the objective $r_i$7, rather than optimizing returns or social welfare directly. This suggests that NePPO should be read as an exploitability-oriented MARL framework rather than as a mere architectural variant of MAPPO or PPO (Kalanther et al., 7 Mar 2026).

6. Limitations, edge cases, and terminological overlap

The paper identifies several limitations. If unilateral incentive differences in the original game are highly non-potential around the target equilibrium, then no useful $r_i$8 may achieve small $r_i$9, and the resulting $\mathcal{N}=\{1,\dots,N\}$00-Nash approximation may be weak. A restricted function class for $\mathcal{N}=\{1,\dots,N\}$01 can cause misspecification, particularly when a shallow parameterization cannot represent equilibrium-relevant potentials. The two-point zeroth-order estimator introduces a bias–variance trade-off through $\mathcal{N}=\{1,\dots,N\}$02 and can destabilize the outer loop when $\mathcal{N}=\{1,\dots,N\}$03 and $\mathcal{N}=\{1,\dots,N\}$04 are poorly tuned. In addition, the method depends on the quality of the inner cooperative and best-response solvers; transient inaccuracies in these components can make the discrepancy estimates misleading before convergence. The paper points to richer potential classes, multi-direction zeroth-order variance reduction, rollout reuse, off-policy evaluation, larger partially observable domains such as SMAC and GRF, CTDE variants with learned $\mathcal{N}=\{1,\dots,N\}$05 critics, and stronger theory under approximate inner solvers as open directions.

A separate source introduces a different use of the name “Near-Potential Policy Optimization.” In the RLVR work on Near-Future Policy Optimization, “NePPO” is described as a direct port of the NPO mechanism, with “near potential” meaning a checkpoint that is close enough to be learnable yet strong enough to improve verified correctness. Under that usage, guide selection maximizes an effective learning signal

$\mathcal{N}=\{1,\dots,N\}$06

where $\mathcal{N}=\{1,\dots,N\}$07 measures verified-correct solutions on the current policy’s failures and $\mathcal{N}=\{1,\dots,N\}$08 is a KL-based proxy for variance induced by distributional shift. The proposed NePPO-select, NePPO-mix, and NePPO-stabilize components are explicitly presented as a semantic reframing of NPO’s trigger–select–replay machinery, not as the general-sum MARL method of (Kalanther et al., 7 Mar 2026). The shared name therefore masks a substantive distinction: the MARL NePPO learns a player-independent potential to approximate Nash equilibria, whereas the RLVR-derived NePPO repurposes near-future checkpoint guidance to balance “strong enough” and “close enough” off-policy signals (Qin et al., 22 Apr 2026).