Dual-Loop Policy Optimization in RL

• Dual-Loop Policy Optimization is a design principle that interleaves a fast inner loop for immediate decisions with a slow outer loop for long-term capability growth.
• It leverages methods like primal-dual updates, two-timescale separation, and planner-learner interactions to balance rapid updates with robust stability.
• Applications span constrained and offline RL, metacognitive systems in multi-agent LLMs, and safe control in LQR, each adapting the dual-loop concept uniquely.

Dual-Loop Policy Optimization denotes a class of reinforcement-learning and agent-learning procedures in which two coupled optimization loops are interleaved. In the most explicit recent usage, it names a metacognitive training scheme that “disentangles immediate decision-making from long-term capability growth”: the inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning from expert feedback (Yang et al., 9 Mar 2026). Closely related structures appear earlier under primal-dual optimization for constrained MDPs (Liang et al., 2018), two-timescale projected gradient descent-ascent for regularized MDPs (Wolter et al., 7 May 2025), dual formulations for policy constraints (Cooman et al., 2024), referential and conservative updates in model-based RL (Zhang, 2022), Dual Policy Iteration (Sun et al., 2018), bi-level offline pessimism (Zhou, 2023), and primal-dual ergodic-risk constrained LQR (Talebi et al., 10 Feb 2025). The shared pattern is a decomposition into two interacting updates that operate on different objects, timescales, or objectives.

1. Scope and recurring structure

The cited literature uses a dual-loop design in several technically distinct ways. In constrained RL, the split is typically between a primal policy update and a dual multiplier update. In regularized LP formulations, it is a fast primal descent and a slow dual ascent. In multi-agent LLM systems, it is an RL loop for “when to ask for help” coupled with a supervised loop for “how to incorporate help.” In planner-learner systems, it is an alternating update between a slow expert policy and a fast reactive policy. In offline RL, it becomes a bi-level interaction between pessimistic value construction and policy improvement.

Framework	First loop	Second loop
APDO (Liang et al., 2018)	On-policy primal policy update	Dual ascent with a one-time off-policy adjustment
PGDA-RL (Wolter et al., 7 May 2025)	Fast update of $V$	Slow update of discounted occupancy $\rho$
HILA DLPO (Yang et al., 9 Mar 2026)	GRPO on deferral actions	Continual learning / SFT on expert demonstrations
DPI (Sun et al., 2018)	Non-reactive expert improvement	Reactive-policy imitation-style update
CDPO (Zhang, 2022)	Referential update under a reference model	Conservative trust-region update
Bi-level offline RL (Zhou, 2023)	Lower-level confidence-set construction	Upper-level conservative policy optimization
ER-LQR (Talebi et al., 10 Feb 2025)	Inner policy optimization over $K$	Outer multiplier update over $\lambda$

A plausible unifying interpretation is that “dual-loop” does not identify a single algorithmic template; it identifies a design principle for nested or alternating optimization. The loops may be primal/dual, planner/learner, RL/SFT, or pessimistic evaluator/policy improver.

2. Lagrangian primal-dual formulations

In constrained Markov Decision Processes, the canonical dual-loop construction starts from a constrained objective. One formulation maximizes the $\gamma$-discounted return

$$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$

subject to long-term cost bounds

$$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$

with Lagrangian

$$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$

and saddle-point objective $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$ (Liang et al., 2018). The inner loop ascends in $\theta$ using an on-policy likelihood-ratio gradient, often through a TRPO/PPO-style surrogate objective

$\rho$0

where $\rho$1 is a Lagrangian advantage estimate (Liang et al., 2018).

A related generic constrained-RL formulation writes

$\rho$2

with

$\rho$3

and interleaved gradient-based updates

$\rho$4

This same framework yields a direct reward-shaping interpretation through

$\rho$5

so the dual variables act as trainable reward modifiers (Cooman et al., 2024).

These formulations establish the most classical sense of dual-loop optimization in RL: the policy is optimized under a fixed multiplier, and the multiplier is updated to enforce feasibility. This suggests that the “dual” in dual-loop often refers literally to Lagrange duality, although later work broadens the pattern beyond multiplier updates.

3. Two-timescale separation and off-policy acceleration

The main algorithmic difficulty in primal-dual methods is that naïve alternation can be sample-inefficient or unstable. “Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning” addresses this in CMDPs by noting that existing methods only use on-policy data for dual updates, which results in sample inefficiency and slow convergence (Liang et al., 2018). APDO keeps the primal update on-policy, but after $\rho$6 iterations performs a one-time jump to an approximately optimal multiplier $\rho$7 estimated offline from a replay buffer. Concretely, it stores every transition in a replay buffer $\rho$8, runs an off-policy primal-dual algorithm on $\rho$9 at $K$0, resets $K$1, and then resumes on-policy dual ascent (Liang et al., 2018).

In the reported MuJoCo point-gather experiment, APDO and CPO both maintain long-term cost at the $K$2 limit, PDO violates the constraint for many epochs, and APDO reaches an average return of $K$3 in $K$4 epochs whereas CPO requires $K$5 epochs; the dual variable exhibits a pronounced jump at $K$6 and then settles near the optimal multiplier $K$7 (Liang et al., 2018). The paper simultaneously states that it does not provide formal convergence proofs or sample-complexity bounds for APDO.

A more theoretically explicit two-timescale construction appears in PGDA-RL. There, a regularized LP reformulation of a finite MDP introduces a primal variable $K$8 and a dual discounted occupancy $K$9, with saddle objective

$\lambda$0

where $\lambda$1 (Wolter et al., 7 May 2025). The key algorithmic choice is a two-timescale stochastic approximation regime: the “fast” variable $\lambda$2 is updated with step size $\lambda$3, the “slow” variable $\lambda$4 with $\lambda$5, and $\lambda$6 (Wolter et al., 7 May 2025). In the online version, the algorithm operates from a single Markov trajectory, uses small replay buffers $\lambda$7 and incoming lists $\lambda$8, updates only visited coordinates, and may choose actions according to

$\lambda$9

to guarantee sufficient exploration (Wolter et al., 7 May 2025).

Under finite-state assumptions, boundedness, Lipschitz gradients, ergodicity of the on-policy chain, strictly positive exploration, and diminishing step sizes satisfying

$\gamma$0

PGDA-RL proves almost-sure convergence

$\gamma$1

and therefore $\gamma$2, without requiring a simulator or a fixed behavioral policy (Wolter et al., 7 May 2025).

Taken together, these works show two distinct reasons for a dual-loop split: acceleration through replay-assisted outer updates, and asymptotic analysis through timescale separation.

4. Metacognitive Dual-Loop Policy Optimization for multi-agent LLMs

In “Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning,” Dual-Loop Policy Optimization is the explicit name of the training mechanism inside the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework (Yang et al., 9 Mar 2026). The metacognitive decision problem is modeled as an MDP whose state $\gamma$3 is a compact “cognitive” embedding of task context $\gamma$4, self context $\gamma$5, peer context $\gamma$6, and optional structured cues $\gamma$7. The action space is

$\gamma$8

and the policy $\gamma$9 outputs a distribution over these actions (Yang et al., 9 Mar 2026).

The inner loop uses GRPO with a cost-aware reward

$$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$0

where $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$1, $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$2, and $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$3 (Yang et al., 9 Mar 2026). At each state, with $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$4 candidate actions, the method forms a reward vector $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$5 and defines centered advantages

$$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$6

The policy-gradient surrogate is

$$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$7

augmented by a KL penalty to a fixed reference policy $$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$8 and an entropy bonus. The full inner-loop loss is

$$R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]$$9

(Yang et al., 9 Mar 2026).

The outer loop is continual learning. Whenever the policy selects $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$0 and receives an expert demonstration $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$1, the system turns that demonstration into a supervised fine-tuning sample with loss

$$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$2

and back-propagates it only on deferral steps (Yang et al., 9 Mar 2026). The combined training schedule maintains replay buffers $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$3 and $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$4, performs GRPO updates from $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$5, performs SFT updates from $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$6, and defines the total loss

$$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$7

(Yang et al., 9 Mar 2026).

The paper’s claims about performance are explicitly empirical. It reports that, as training progresses from the random meta-policy to GRPO and then to full DLPO, the system moves toward higher task accuracy while simultaneously reducing the DEFER rate; GRPO alone rapidly learns to avoid overly frequent deferral but plateaus in accuracy, whereas adding the continual-learning loop yields further accuracy gains while deferral continues to shrink (Yang et al., 9 Mar 2026). It also states that the inner-loop KL penalty and entropy bonus resemble TRPO/PPO-style stabilization, but it does not provide a formal theorem on convergence rates (Yang et al., 9 Mar 2026).

5. Planner-learner and conservative trust-region variants

A different lineage of dual-loop optimization does not center on dual variables at all. “Dual Policy Iteration” maintains two stationary policies at each iteration: a reactive policy $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$8, typically a parametric function approximator deployed at test time, and a non-reactive “expert” policy $$C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,$$9, typically a slower planner such as local LQR or tree search (Sun et al., 2018). The overall viewpoint is

$$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$0

whose unique Nash equilibrium is $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$1 (Sun et al., 2018). The two loops alternate as follows: first, fit a local model $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$2 from rollout data; second, update $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$3 by model-based optimal control under a trust-region constraint around $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$4; third, update $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$5 by minimizing the expert’s local disadvantage under its own trust region (Sun et al., 2018). The paper gives a convergence analysis extending existing approximate policy-iteration theory and notes that, when $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$6, the method reduces to CPI/TRPO, while forward-search experts recover ExIt or AlphaGo-Zero-style schemes (Sun et al., 2018).

“Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning” uses yet another two-stage construction (Zhang, 2022). At each iteration, it first performs a Referential Update under a reference model $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$7, for instance the least-squares estimate $$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$8, by setting

$$\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,$$9

It then performs a Conservative Update

$$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$0

or, in practice, a KL-based variant via Pinsker’s inequality (Zhang, 2022). This split is dual-loop in the sense of first producing a stable reference-improving policy $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$1, then performing a conservative posterior-expectation update to $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$2.

The theoretical profile of CDPO is unusually explicit. If a posterior-sampling algorithm satisfies $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$3, then CDPO satisfies

$$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$4

Under the stated regularity conditions it also establishes a monotonic policy-improvement result and a global-optimality result

$$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$5

matching the $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$6 rate of PSRL/OFU up to complexity factors (Zhang, 2022). The cited discussion contrasts this with OFU and PSRL: OFU can over-explore when the confidence set is large, while PSRL can make aggressive jumps when sampled models fluctuate; CDPO uses a stable reference and an expectation-based conservative update to mitigate those effects (Zhang, 2022).

These planner-learner and trust-region variants broaden the meaning of dual-loop optimization. Here the two loops are not primal and dual variables; they are two policy-improvement operators with different stability and exploration roles.

6. Bi-level pessimism, ergodic-risk control, and theoretical limits

In offline RL, a dual-loop or bi-level structure appears as a hierarchy between policy improvement and pessimistic value construction. “Bi-Level Offline Policy Optimization with Limited Exploration” first builds, for a fixed policy $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$7, a confidence set of $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$8-functions whose weighted Bellman errors are small: $$\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)$$9 where $\theta$0 constrains a nonnegative weighting function $\theta$1 through a detection penalty $\theta$2 (Zhou, 2023). The upper level then chooses

$\theta$3

where $\theta$4 is the most pessimistic value estimate in the confidence set (Zhou, 2023). The practical solver introduces a penalized saddle-point loss

$\theta$5

alternates proximal updates on $\theta$6 and closed-form updates on $\theta$7, and then updates the policy by a mirror-descent step with negative-entropy divergence (Zhou, 2023). Its theoretical guarantee is explicitly “best-effort”: under realizability and boundedness, and without coverage or completeness assumptions, the learned policy satisfies the stated regret bound against any comparator policy (Zhou, 2023).

A more classical control-theoretic dual-loop appears in ergodic-risk constrained LQR. There the system is

$\theta$8

with stationary linear feedback $\theta$9, average cost

$\rho$00

and ergodic-risk constraint

$\rho$01

based on the asymptotic conditional variance of the uncertainty increment $\rho$02 (Talebi et al., 10 Feb 2025). The constrained optimization problem is

$\rho$03

with Lagrangian

$\rho$04

For fixed $\rho$05, the policy gradient has closed form: $\rho$06 (Talebi et al., 10 Feb 2025). The algorithm performs an inner loop over $\rho$07 until $\rho$08, then an outer projected ascent

$\rho$09

Under the stated assumptions, the inner loop converges quadratically via Riemannian Newton or linearly via gradient descent, the outer loop converges at rate $\rho$10 in the duality gap, and the overall complexity is $\rho$11 (Talebi et al., 10 Feb 2025). In the reported Grumman X-29 experiment with Student-$\rho$12 process noise, ER-LQR achieves target risk $\rho$13 while incurring only $\rho$14 increase in $\rho$15 (Talebi et al., 10 Feb 2025).

The literature therefore places strong limits on any attempt to treat dual-loop policy optimization as a single, fully unified theory. Some instances provide almost-sure convergence to a unique regularized optimum (Wolter et al., 7 May 2025); some provide monotonic-improvement or regret guarantees (Zhang, 2022, Sun et al., 2018, Talebi et al., 10 Feb 2025, Zhou, 2023); some explicitly report the absence of formal convergence proofs (Liang et al., 2018); and some provide empirical rather than theorem-level evidence (Yang et al., 9 Mar 2026). A common misconception is that dual-loop necessarily means a Lagrange-multiplier method. The cited work instead shows several non-equivalent meanings: multiplier enforcement in CMDPs, timescale separation in regularized LPs, planner-learner alternation in approximate policy iteration, conservative trust-region refinement in model-based RL, and RL/SFT coupling in human-in-the-loop multi-agent LLM systems.