Bilevel Optimization over Saddle Points of Zero-Sum Markov Games

Published 26 May 2026 in cs.LG, cs.AI, math.OC, and stat.ML | (2605.26654v1)

Abstract: Reinforcement learning (RL) often has a hierarchical structure, where an upper-level (UL) learner selects model parameters and a lower-level (LL) decision-making process responds, naturally leading to a bilevel optimization problem. Most existing bilevel RL methods assume a single-policy LL Markov decision process (MDP), and therefore fail to capture competitive structures arising in applications such as incentive design, where multiple policies interact. We study bilevel optimization problems in which the LL problem is a regularized min-max zero-sum Markov game and the UL objective is optimized through the saddle-point equilibrium induced by the LL game. In this work, we propose penalty-augmented Nikaido-Isoda descent-ascent (PANDA), a penalty-based first-order policy-gradient method based on the Nikaido-Isoda function. By exploiting the min-max game structure, PANDA avoids computing UL hypergradients and does not require second-order information. We prove that PANDA converges to stationary points without convexity assumptions on either the UL or LL objectives. Moreover, PANDA reaches an $ε$-stationary point in $\tilde{\mathcal{O}}(ε^{-1})$ iterations with sample complexity $\tilde{\mathcal{O}}(ε^{-3})$, matching the best-known rates for bilevel RL with single-policy LL MDPs. Experiments demonstrate the superior performance of PANDA over closely related baselines.

Abstract PDF Upgrade to Chat

Authors (3)

Summary

The paper introduces the PANDA algorithm, a penalty-augmented NI descent–ascent method for resolving bilevel reinforcement learning with LL saddle-point Markov games.
It establishes near-optimal convergence, demonstrating an ε-stationary point in O(ε⁻¹) iterations with a sample complexity of O(ε⁻³) even in nonconvex–nonconcave setups.
Empirical results in synthetic tasks and gridworld environments validate PANDA's superior performance over baselines while effectively enforcing the LL Nash equilibrium.

Bilevel Optimization over Markov Game Saddle Points: Summary and Technical Analysis

The paper "Bilevel Optimization over Saddle Points of Zero-Sum Markov Games" (2605.26654) introduces a rigorous framework and algorithmic advances for bilevel reinforcement learning (BRL) when the lower-level (LL) problem is a regularized min–max zero-sum Markov game (MMZSMG), addressing a gap in prior work which mostly considers single-agent MDPs in the LL. The work proposes the penalty-augmented Nikaido–Isoda descent–ascent (PANDA) algorithm, a policy-gradient-based bilevel optimizer designed to handle the coupled nonconvex–nonconcave structure of multi-policy zero-sum games in the LL, and establishes near-optimal convergence and sample complexity guarantees in the stochastic regime.

Problem Formulation and Motivation

Bilevel RL models environments in which an upper-level (UL) controller selects environment or incentive parameters while the LL comprises a learning process—here, a two-player zero-sum Markov game—whose equilibrium response is a saddle point of a min–max objective. Unlike conventional BRL, which focuses on single-policy MDPs at the LL and therefore admits more direct optimization techniques, the min–max saddle point structure forces a strategic coupling: both LL agents adjust their nonstationary policies in response to the environment and each other, rendering hypergradient estimation intractable, especially in the sample-based (stochastic) regime.

Key properties of the MMZSMG setting analyzed:

The LL admits a unique regularized Nash equilibrium (NE) due to the strong convexity–concavity of the regularization (typically entropy/KL).
The UL objective is evaluated at the LL saddle-point equilibrium and is a generally nonconvex function of both UL parameters and the induced NE policies.

Algorithmic Contributions: The PANDA Method

PANDA is a first-order, double-loop, stochastic optimization method that leverages the Nikaido–Isoda (NI) gap to enforce LL equilibrium stationarity via a penalty construction, thereby circumventing the need for Hessian-based hypergradients or full implicit differentiation. The approach comprises:

Best-Response Approximation: Using stochastic policy gradient ascent/descent steps to approximately solve the inner best-response problems defining the NI gap.
Penalty-Subproblem Optimization: Alternating policy gradient-based updates to LL policy parameters on the penalized surrogate objective combining the UL loss and the LL NI gap.
Hypergradient Step: Updating the UL parameters via a stochastic gradient estimate of the penalized single-level objective, based on the outcome of the LL penalized subproblem.

This structure is compatible with entropy/KL-regularized policy parameterizations and admits scalable Monte Carlo implementations based on trajectory samples.

Figure 1: Performance of PANDA versus environment sample steps: (Left) UL incentive reward; (Right) LL NE gap.

Theoretical Results: Convergence Without Convexity

The principal theoretical contribution is the demonstration that PANDA achieves an $\varepsilon$ -stationary point of the original bilevel problem in $\tilde{\mathcal{O}}(\varepsilon^{-1})$ iterations with sample complexity $\tilde{\mathcal{O}}(\varepsilon^{-3})$ , where $\tilde{\mathcal{O}}$ hides logarithmic terms. These rates align with what is achievable in BRL with a single-policy LL under the Polyak–Łojasiewicz (PL) condition. Remarkably, this is achieved without strong convexity/concavity assumptions on either the UL or the LL objectives and in the stochastic, trajectory-sampled setting.

Key technical properties enabling these results:

Proved strong structural properties for the NI function: non-uniform PL condition and Lipschitz continuity, which facilitate error-control and convergence analysis.
Exploitation of the regularization-induced uniqueness of NE in MMZSMGs, enabling tractable penalization and dual-loop descent–ascent.
Avoidance of hypergradient computation—no Hessian inversion or expensive second-order steps required.

Figure 2: Ablation study: Effect of penalty parameter $\lambda$ on (Left) UL incentive reward, (Right) LL NE gap.

Empirical Validation

Experiments demonstrate the superiority of PANDA over several recent competitive baselines, including heuristic meta-gradients and penalty-based approaches that either require strong side-information or lack theoretical guarantees:

On a synthetic incentive-design task involving a randomly parameterized MMZSMG, PANDA consistently attains higher UL reward and lower NE gap than alternatives, closely matching an oracle with access to dynamic programming solutions (Figure 1).
A detailed ablation (Figure 2) confirms the tradeoff controlled by the penalty parameter $\lambda$ : smaller $\lambda$ trades off slower convergence to NE for faster UL progress, while larger $\lambda$ enforces more accurate LL equilibrium at the cost of UL reward suboptimality.
In the complex Sentinel-Intruder environment (Figures 3 and 4), including high-dimensional gridworlds, PANDA achieves a lower UL loss (e.g., minimizing the sentinel's visits to restricted areas) while guaranteeing the LL equilibrium constraint is tightly enforced, with robust performance as the problem scales.

Figure 3: PANDA performance on the Sentinel-Intruder game: (Left) $5 \times 5$ grid UL loss; (Right) $20 \times 20$ grid UL loss.

Figure 4: Sentinel-Intruder game schematic—sentinel seeks to intercept while avoiding yellow restricted regions.

Theoretical and Practical Implications

The formal and empirical findings have several implications for theory and applications of multi-agent RL:

Scalability: The established sample complexity allows practical deployment of bilevel optimization for problems with large state/action spaces and with coupled multi-agent equilibria.
Generalization: The penalty-augmented NI gap framework can be ported to other regularized multi-agent game-theoretic RL and RLHF settings, including preference learning for LLM alignment.
Algorithmic Modularity: By structuring the LL update as a generic penalty-subproblem, PANDA can flexibly incorporate more advanced off-policy/batch RL oracles for further scalability.
Limitations and Extensions: While current results require regularization for uniqueness and smoothness, it is plausible that future developments can relax this to broader classes of multi-agent games (e.g., general-sum, non-regularized).

Future Outlook

Potential research avenues opened by this work include:

Extending the penalty-based NI descent–ascent methodology to general-sum Markov games and nonzero-sum multi-agent RL.
Devising single-loop PANDA-like algorithms with reduced variance and sample efficiency, using advances in variance reduction for stochastic policy gradients.
Applying this bilevel optimization framework to RLHF and mechanism design with multiple learning agents and equilibria.

Conclusion

This paper resolves several open technical challenges in bilevel RL involving zero-sum Markov games by introducing a first-order, sample-based algorithm with robust theoretical guarantees and validated empirical superiority. It broadens the applicability of BRL to realistic multi-agent and adversarial environments, setting the stage for further advances in scalable multi-agent reinforcement and incentive design.

Figure 5: $\tilde{\mathcal{O}}(\varepsilon^{-1})$ 0 grid setup for the Sentinel-Intruder game—the environment for the scalability experiments.

Figure 6: $\tilde{\mathcal{O}}(\varepsilon^{-1})$ 1 grid Sentinel-Intruder: (Left) UL loss versus trajectories; (Right) LL NE gap versus trajectories.

Figure 7: $\tilde{\mathcal{O}}(\varepsilon^{-1})$ 2 grid Sentinel-Intruder: (Left) UL loss versus trajectories; (Right) LL NE gap versus trajectories.