Penalty-Augmented NI Descent-Ascent (PANDA)

• The PANDA framework introduces a penalty-based formulation that transforms hierarchical equilibrium constraints into a smooth single-level optimization using the Nikaido-Isoda gap.
• It applies first-order policy-gradient methods to solve regularized min-max Markov games and multi-leader-follower problems without requiring higher-order derivative computations.
• Empirical evaluations demonstrate PANDA’s capability to achieve near-oracle performance and enforce lower-level feasibility with provable convergence guarantees.

Penalty-Augmented Nikaido-Isoda Descent-Ascent (PANDA) is a penalty-based first-order framework for hierarchical optimization and game-theoretic equilibrium constraints that uses the Nikaido-Isoda gap, or a regularized Nikaido-Isoda value function, to convert lower-level equilibrium conditions into a smooth single-level objective. In "Bilevel Optimization over Saddle Points of Zero-Sum Markov Games" (Zheng et al., 26 May 2026), PANDA is introduced as a first-order policy-gradient method for bilevel optimization whose lower level is a regularized min-max zero-sum Markov game. In the regularized Nikaido-Isoda reformulation of "A Regularized Nikaido-Isoda Function Approach to Multi-Leader-Follower Games" (Hori et al., 18 Jun 2026), the same penalty-augmented Nikaido-Isoda logic yields a descent-ascent scheme for multi-leader-follower games that avoids derivative information on the followers' game and does not assume convexity of each follower's problem.

1. Problem classes and conceptual scope

PANDA is designed for problems in which an upper-level decision must respect a lower-level equilibrium. The two settings documented in the cited sources are structurally distinct but algorithmically aligned. One is bilevel optimization in reinforcement learning, where upper-level parameters $x$ interact with a lower-level saddle-point problem over policy parameters $(\phi,\psi)$. The other is the multi-leader-follower game (MLFG), where leaders $x=(x^1,\dots,x^N)$ compete at the upper level while accounting for followers’ equilibrium responses $y=(y^1,\dots,y^M)$ (Zheng et al., 26 May 2026).

In the RL formulation, the problem is

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$

Here $x$ are upper-level parameters, $\phi,\psi$ parameterize two policies $\pi_\phi,\pi_\psi$ in a zero-sum Markov game, $f(x,\phi,\psi)$ is the upper-level objective, and $J(x,\phi,\psi)$ is the regularized lower-level value function (Zheng et al., 26 May 2026).

In the MLFG formulation, leaders solve

$(\phi,\psi)$0

with $(\phi,\psi)$1 constrained to be a Nash equilibrium of the followers’ game. The associated regularized Nikaido-Isoda construction replaces this hierarchical dependence by a single-level differentiable Nash equilibrium problem in extended variables $(\phi,\psi)$2 (Hori et al., 18 Jun 2026).

Setting	Lower-level object	PANDA role
Bilevel RL	Regularized min-max zero-sum Markov game	First-order policy-gradient method on a penalty reformulation
MLFG	Followers’ Nash equilibrium encoded by a regularized NI function	Descent-ascent scheme on a $(\phi,\psi)$3-penalized NEP

A unifying interpretation is that PANDA treats equilibrium satisfaction as a penalized optimality gap rather than as an implicit-function or KKT system. This suggests a deliberate shift away from formulations that require higher-order sensitivity information.

2. Nikaido-Isoda penalty construction

The defining object in PANDA is the Nikaido-Isoda gap, or its regularized variant, used as a feasibility surrogate for lower-level equilibrium.

In the RL setting, the classical NI gap is

$(\phi,\psi)$4

It satisfies $(\phi,\psi)$5 and $(\phi,\psi)$6 is lower-level optimal, i.e., a saddle point. The bilevel problem is then rewritten as

$(\phi,\psi)$7

and a standard penalty objective is formed: $(\phi,\psi)$8 For $(\phi,\psi)$9, the minimizers of $x=(x^1,\dots,x^N)$0 recover solutions of the bilevel problem, while in practice one picks $x=(x^1,\dots,x^N)$1 large but finite (Zheng et al., 26 May 2026).

In the MLFG setting, the regularized Nikaido-Isoda function is

$x=(x^1,\dots,x^N)$2

with value function

$x=(x^1,\dots,x^N)$3

Under the standing smoothness assumptions, for every $x=(x^1,\dots,x^N)$4 the maximizer

$x=(x^1,\dots,x^N)$5

is unique, and $x=(x^1,\dots,x^N)$6 is continuously differentiable on $x=(x^1,\dots,x^N)$7. The leader-wise penalty-augmented objective is

$x=(x^1,\dots,x^N)$8

which defines a $x=(x^1,\dots,x^N)$9-penalized Nash equilibrium problem in the extended variables $y=(y^1,\dots,y^M)$0 (Hori et al., 18 Jun 2026).

The significance of these constructions is methodological. The lower-level equilibrium is encoded as a nonnegative gap function, so feasibility is enforced by driving that gap to zero. In RL the gap is the saddle-point NI gap $y=(y^1,\dots,y^M)$1; in MLFG the role is played by the regularized value function $y=(y^1,\dots,y^M)$2.

3. Algorithmic structure

In the RL formulation, PANDA is a triple-loop policy-gradient method operating on $y=(y^1,\dots,y^M)$3. The inner “best-response” loop computes approximate max and min of $y=(y^1,\dots,y^M)$4 and $y=(y^1,\dots,y^M)$5 to estimate the NI gap and its gradients. The middle “penalty-subproblem” loop updates $y=(y^1,\dots,y^M)$6 to approximately minimize the penalized objective. The outer loop updates $y=(y^1,\dots,y^M)$7 by gradient descent on $y=(y^1,\dots,y^M)$8 (Zheng et al., 26 May 2026).

The key gradient identities are given in terms of the on-policy $y=(y^1,\dots,y^M)$9-function

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$0

Lemma 3.1 states that

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$1

with an analogous formula for $$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$2, and Lemma 3.2 gives

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$3

These formulas enable PANDA to use only first-order policy-gradient oracles.

In the MLFG formulation, PANDA alternates three operations. First, an ascent step computes

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$4

which may be solved by projected gradient-ascent or any smooth solver over the convex set $$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$5. Second, a descent step forms

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$6

where $$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$7 is obtained via Danskin’s theorem as $$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$8, and then updates

$$\min_{x\in\mathbb{R}^n,\phi\in\mathbb{R}^p,\psi\in\mathbb{R}^q} f(x,\phi,\psi) \quad \text{subject to} \quad (\phi,\psi)\in \arg\min_{\phi'}\max_{\psi'} J(x,\phi',\psi').$$9

Third, the penalty parameter is increased when $x$0 is not small enough, for example through

$x$1

In both settings, the operative pattern is the same: compute a response certificate for the lower level, differentiate the penalized surrogate, and update the upper-level and lower-level variables using first-order steps (Hori et al., 18 Jun 2026).

4. Theoretical guarantees

For bilevel optimization over regularized zero-sum Markov games, the main convergence theorem states that under standard bounded-variance and smoothness assumptions, and with the prescribed step sizes and $x$2, after $x$3 outer iterations,

$x$4

With the choices

$x$5

one obtains

$x$6

Accordingly, PANDA reaches an $x$7-stationary point in $x$8 iterations with sample complexity $x$9 (Zheng et al., 26 May 2026).

For the MLFG regularized NI reformulation, the convergence claims are stated under Assumptions 3.1 and 3.2, together with standard diminishing step-size conditions $\phi,\psi$0 and $\phi,\psi$1. Every accumulation point $\phi,\psi$2 of $\phi,\psi$3 is a variational equilibrium of the $\phi,\psi$4-penalized NEP. If $\phi,\psi$5, then

$\phi,\psi$6

so any limit point satisfies $\phi,\psi$7 and hence is a leader-follower Nash equilibrium of the original MLFG. Under the Kurdyka-Łojasiewicz property of subanalytic $\phi,\psi$8, the entire trajectory has finite length and converges to a single limit. Under Lipschitz continuity of the gradients, one also obtains an $\phi,\psi$9 bound on the smallest stationarity measure $\pi_\phi,\pi_\psi$0 (Hori et al., 18 Jun 2026).

These guarantees show that PANDA is not merely a heuristic penalty method. In both settings, the penalty mechanism is coupled to explicit stationarity and feasibility results, although the exact statements depend on the surrounding problem class.

5. Relation to EPEC, implicit differentiation, and hypergradient methods

A central feature of PANDA is that it avoids the derivative burdens associated with more classical reformulations. In the MLFG literature, a typical approach reformulates the problem as an equilibrium problem with equilibrium constraints by replacing the lower-level game with its KKT conditions, while another approach substitutes unique follower response functions into each leader’s problem. Both reformulations may lack scalability since higher-order derivatives may be required when solving the resulting problems. The regularized Nikaido-Isoda formulation is presented precisely to avoid that dependence: it neither requires derivative information on the followers’ game nor assumes convexity of each follower’s problem (Hori et al., 18 Jun 2026).

The RL version makes the same design choice in a different language. By exploiting the min-max game structure, PANDA avoids computing upper-level hypergradients and does not require second-order information. The paper emphasizes that the method uses only first-order policy-gradient oracles, with no hypergradient or Hessian inverses (Zheng et al., 26 May 2026).

Two clarifications follow from the stated results. First, PANDA is not an exact reformulation at arbitrary finite penalty. In the RL formulation, recovery of bilevel solutions is tied to the regime $\pi_\phi,\pi_\psi$1, while practical implementations use $\pi_\phi,\pi_\psi$2 large but finite. Second, in the MLFG formulation, exact leader-follower equilibrium is linked to the condition $\pi_\phi,\pi_\psi$3, and the convergence statements invoke either equilibria of the penalized game or the regime $\pi_\phi,\pi_\psi$4. A common misconception would therefore be to equate penalty minimization immediately with exact lower-level feasibility without the accompanying limiting or stationarity arguments.

6. Empirical behavior and practical significance

The RL paper reports empirical evaluation on two classes of problems. The first is a synthetic incentive-design problem in which the lower level is a two-player zero-sum Markov game parameterized by incentive $\pi_\phi,\pi_\psi$5, while the upper level maximizes a designer’s reward in a base MDP subject to lower-level Nash equilibrium. The reported comparisons are PANDA versus Meta-Gradient, Differentiable Arbitrating (DA), and PBRL. The metrics plotted are upper-level reward and lower-level NI-gap versus environment steps, and the stated result is that PANDA achieves near-oracle upper-level performance, closes the NI gap to zero, and outperforms baselines (Zheng et al., 26 May 2026).

The second reported environment is the Sentinel-Intruder grid-world, where the sentinel and intruder play a zero-sum chase game on a $\pi_\phi,\pi_\psi$6 and $\pi_\phi,\pi_\psi$7 grid with restricted cells penalized by the upper level. The upper level learns $\pi_\phi,\pi_\psi$8 to discourage the sentinel from restricted regions, while the lower level reacts through a Nash equilibrium between intruder and sentinel. The implementation uses small CNNs plus fully connected layers for $\pi_\phi,\pi_\psi$9 and the reward model $f(x,\phi,\psi)$0, Monte Carlo gradients with batch size $f(x,\phi,\psi)$1, $f(x,\phi,\psi)$2 inner iterations, and regularizer $f(x,\phi,\psi)$3. The reported result is that PANDA drives upper-level loss lower and enforces lower-level equilibrium more reliably across both small and large grids (Zheng et al., 26 May 2026).

For the MLFG regularized NI formulation, the significance is primarily structural rather than benchmark-driven. The single-level nature and differentiability of $f(x,\phi,\psi)$4 render first-order methods easily scalable to large $f(x,\phi,\psi)$5, and no second derivatives of the followers’ cost functions $f(x,\phi,\psi)$6 or constraint Jacobians are needed (Hori et al., 18 Jun 2026). This suggests that PANDA is best understood not as a domain-specific RL algorithm alone, but as a broader penalty-augmented Nikaido-Isoda strategy for equilibrium-constrained optimization in which smooth first-order updates replace direct higher-order sensitivity calculations.