Bi-Level Policy Gradient Algorithm

• Bi-Level Policy Gradient is a framework that decomposes reinforcement learning into an upper-level objective and a lower-level policy optimization to address nested objectives and constraints.
• It employs tailored gradient estimators, log-barrier penalties, and natural-gradient updates to manage challenges such as Blackwell optimality, Nash equilibria, and reward design.
• Practical implementations demonstrate scalable, sample-efficient convergence across multi-agent bidding, actor–critic setups, and synthetic control tasks, backed by rigorous theoretical guarantees.

The Bi-Level Policy Gradient (BPG) Algorithm is an optimization paradigm for reinforcement learning and multi-agent decision processes where policy search is naturally decomposed into two levels: an upper-level objective interacting with a lower-level policy optimization or equilibrium computation. The BPG framework encapsulates a diverse set of approaches for bilevel-structured policy optimization in Markov Decision Processes (MDPs), actor–critic architectures, and constrained multi-agent learning, leveraging tailored gradient estimators, penalty reformulations, and natural-gradient or hypergradient methodologies. It is conceptually distinct from standard single-level policy gradient algorithms due to its handling of nested objectives and constraints, arising in problems such as Blackwell optimality, Nash equilibrium bidding, Stackelberg games, and reward design.

1. Bilevel Policy Optimization: Problem Structure

A general bilevel policy optimization program is expressed as:

$$\min_{x \in \mathbb{R}^d} \Phi(x) = f(x, \pi^*(x)), \quad \text{s.t. } \pi^*(x) \in \arg\max_\pi J(x, \pi),$$

where $x$ parameterizes the upper-level objective $f$ and the reward $r_x$ of the lower-level MDP. The lower-level task is standard policy optimization (maximize $J$), while the upper-level objective seeks to optimize a functional of the best-response policy. This pattern is found in:

• Near-Blackwell-optimal RL, where the lower level maximizes steady-state (gain) and the upper level secondary criteria (bias) over gain-optimal polices (Dewanto et al., 2021).
• Multi-agent reinforcement learning with Nash equilibrium-style constraints, such as social welfare maximization given approximate equilibrium compliance (Mou et al., 13 Mar 2025).
• Actor–critic (AC) algorithms viewed through a Stackelberg/bilevel lens: the actor’s policy is optimized subject to the critic providing a best response (Prakash et al., 16 May 2025).
• Reward design and constrained RL, where upper-level parameters define environment or reward shaping for a subordinate agent (Zeng et al., 23 Jan 2026).

Formally, bilevel RL problems are nonconvex and may involve nonsmooth and set-valued mappings due to nonuniqueness in optimal lower-level policies, requiring careful regularization or constraint relaxation mechanisms.

2. Canonical BPG Algorithms and Theoretical Properties

a. Gain-then-Bias BPG for Blackwell Optimality

The algorithm in "A nearly Blackwell-optimal policy gradient method" (Dewanto et al., 2021) addresses the classic problem of maximizing average reward (gain) and then, among gain-optimal policies, maximizing the bias for improved transient performance. The explicit bilevel objective is:

• Lower level: $\Pi^*_g = \arg\max_\theta v_g(\theta)$, where $v_g$ is the gain.
• Upper level: $$\theta^*_b \in \arg\max_{\theta\in\Theta^*_g} v_b(\theta, s_0)$$, the bias at initial state $s_0$.

This is enforced via a logarithmic barrier to maintain the gain constraint, yielding the barrier-augmented objective:

$$\mathcal{B}(\theta) = v_b(\theta, s_0) + \beta \log(v_g(\theta) - v^*_g + \zeta), \quad \beta > 0.$$

The BPG updates alternate between gain maximization ($\nabla v_g \approx 0$) and compensatory bias ascent subject to the gain constraint, using natural-gradient style updates with Fisher preconditioning (Dewanto et al., 2021).

b. Nash Equilibrium Constrained BPG

In large-scale multi-agent settings, such as auto-bidding, the Nash-Equilibrium Constrained Bidding (NCB) problem is solved with BPG by converting the equilibrium constraint ($\epsilon$-Nash) into a penalty term and exploiting permutation-equivalence to reduce complexity (Mou et al., 13 Mar 2025). The primal-dual method optimizes a penalized Lagrangian:

$$L(\theta, \lambda, \nu) = \sum_{i=1}^N \left[ \lambda_i G_i(\nu_i; \theta) - (1+\lambda_i) G_i(\theta) - \lambda_i \epsilon \right]$$

with $\theta$ as the shared parameter and $\nu_i$ denoting unilateral deviations. The penalty and unified optimization restructure the problem for scalability, ensuring convergence to an $\epsilon$-NE feasible policy. Theoretical results guarantee that critical points of the penalized BPG problem are approximate solutions to the true bilevel NCB objective under mild assumptions (Mou et al., 13 Mar 2025).

c. Single-Loop Penalty and Regularized Actor-Critic BPG

"Regularized Actor-Critic Algorithm for Bi-Level Reinforcement Learning" (Zeng et al., 23 Jan 2026) addresses computational inefficiency in nested-loop bilevel methods by introducing a penalty reformulation and vanishing entropy regularization:

$$\mathcal{L}_{w,\tau}(x, \pi) = f(x, \pi) + \frac{1}{w}(J_\tau(x, \pi^*_\tau(x)) - J_\tau(x, \pi))$$

where $J_\tau$ adds entropy to the lower-level RL objective, $w>0$ is a penalty vanishing to zero. Both upper-level ($x$) and lower-level policy updates are handled in a single stochastic loop, and the upper-level gradient is estimated with first-order quantities only. The approach enjoys finite-sample convergence guarantees under Polyak-Lojasiewicz-like conditions, with no requirement of solving the lower-level RL problem exactly at each step (Zeng et al., 23 Jan 2026).

d. Hypergradient BPG for Actor–Critic via Nyström Approximation

In the actor–critic context, the actor's update must account for the critic's best-response dependence, leading to a hypergradient:

$$\partial_\theta F(\theta) = -\partial_\theta J_a(\theta, \phi^*) + \partial_\phi J_a(\theta, \phi^*) [\partial^2_{\phi\phi} L_c(\phi^*; \theta)]^{-1} \partial^2_{\theta\phi} L_c(\phi^*; \theta)$$

where hypergradient computation is intractable for high-dimensional $\phi$. BPG here uses the Nyström method to approximate the required inverse-Hessian-vector product efficiently, ensuring that hypergradient steps converge to Stackelberg equilibria in polynomial time, assuming linearity in the critic (Prakash et al., 16 May 2025).

3. Practical Algorithms and Sampling Schemes

BPG algorithms commonly use the following practical components:

• Score function estimators for unbiased policy gradients in both upper and lower levels (Dewanto et al., 2021, Zeng et al., 23 Jan 2026).
• Fisher information matrix preconditioning for natural gradient steps, separately computed for each objective (Dewanto et al., 2021).
• Mini-batch approximations and mixing-based phase separation, accumulating gradients during pre-mixing with steady-state stochastic correction post-mixing (Dewanto et al., 2021).
• Single- and nested-loop architectures: traditional methods require an inner optimization loop (nested actor–critic), but penalty-based BPG achieves single-loop sampling with multi-timescale step sizes to ensure proper tracking and separation of timescales (Zeng et al., 23 Jan 2026).
• Penalty and entropy weights with decaying schedules to ensure asymptotic convergence to the original (unregularized) bilevel problem, with recommended decay exponents and step-size orderings detailed in the literature (Zeng et al., 23 Jan 2026).

4. Theoretical Guarantees and Complexity

BPG methods are supported by several theoretical insights:

• Near-Blackwell optimality: In finite unichain MDPs, any stationary deterministic policy may be nearly-Blackwell-optimal; the log barrier ensures convergence in the limit $\beta \to 0$ (Dewanto et al., 2021).
• Primal–dual convergence: The NCB-BPG approach guarantees convergence to a saddle-point solution under standard Lagrangian arguments, and uniform penalization of the lower-level constraint preserves feasibility (Mou et al., 13 Mar 2025).
• Complexity analysis: For the regularized, penalty-based BPG, the single-loop version achieves sample complexity $\tilde O(\epsilon^{-10})$ for unregularized objectives and $\tilde O(\epsilon^{-3})$ for fixed-entropy regularization, matching nested-loop methods (Zeng et al., 23 Jan 2026).
• Scalability: In multi-agent systems, permutation-equivariant structure ensures that the complexity of the BPG algorithm is independent of the number of agents, as only two gradient blocks need be computed irrespective of $N$ (Mou et al., 13 Mar 2025).
• Hypergradient error: The Nyström approximation ensures that the inverse-Hessian-vector product required for the actor hypergradient can be computed at $\mathcal O(p r^2 + r^3)$ cost, with provable error bounds (Prakash et al., 16 May 2025).

5. Empirical Results and Applications

Empirical validation of BPG strategies spans several domains:

• GridWorld tabular MDPs and RLHF-driven language modeling for the penalty-based, single-loop BPG, demonstrating finite-time convergence and manageability of the multi-timescale update scheme (Zeng et al., 23 Jan 2026).
• Blackwell-optimal policy search in synthetic and continuous Markov chains, with the gain-then-bias BPG revealing improved transient performance over standard discounted reward optimizers (Dewanto et al., 2021).
• Multi-agent ad-auction environments, where NCB-BPG exacts social welfare optimization subject to equilibrium guarantees without scaling costs, outperforming independent or heuristically-constrained baselines (Mou et al., 13 Mar 2025).
• Standard continuous and discrete RL control tasks (Gym, Brax) used to benchmark hypergradient-based BLPO against PPO and conjugate-gradient variants, consistently matching or exceeding PPO performance and demonstrating robustness to conditioning issues in the critic's loss (Prakash et al., 16 May 2025).

6. Comparative Overview of BPG Variants

Approach	Problem Structure	Core Technique
Gain-then-bias BPG (Dewanto et al., 2021)	Single-agent, Blackwell/bias	Log-barrier, split-phase nat.grad
NCB-BPG (Mou et al., 13 Mar 2025)	Multi-agent, NE-constrained	Penalized, permutation-equivariant
Single-loop BPG (Zeng et al., 23 Jan 2026)	Upper-level reward design	Entropy-penalty, single-loop AC
BLPO (Prakash et al., 16 May 2025)	Actor–critic Stackelberg	Nyström hypergradient, nested critic

The unifying feature is the structured exploitation of problem-specific bilevel structure, either through explicit constraints (log-barrier/penalty), tailored gradient estimators, or architectural reduction (single vs. nested loop), always ensuring policy parameter updates account for the lower-level best response or equilibrium condition.

7. Significance and Implications

The BPG framework enables the principled resolution of policy search tasks in reinforcement learning and multi-agent domains where objectives are inherently compositional or constrained. Established methods in RL, such as single-level policy gradients, fail to address these structures, especially where secondary criteria (bias, welfare, constraint satisfaction) or interdependencies (actor/critic, equilibrium) are fundamental. BPG yields provably consistent, scalable, and sample-efficient algorithmic approaches:

• It facilitates natural gradients across bilevel objectives and enables efficient, unbiased hypergradient estimation without explicit second-order information.
• In multi-agent and economic systems, permutation-equivariant reductions render policy optimization tractable at population scales.
• Penalty and barrier methods permit flexible constraint satisfaction (Blackwell optimality, NE compliance) and allow seamless balancing of multiple objectives.
• Theoretical rates match or exceed traditional nested-loop approaches while single-loop sampling improves practical sample efficiency.

A plausible implication is that BPG-style approaches will increasingly underpin complex RL systems, from structured reward design to large-scale equilibria in markets and multi-agent learning, with ongoing research directed at generalizing the penalty, reduction, and hypergradient schemes to broader classes of bilevel and compositional optimization problems in reinforcement learning.