---
title: Dual-Loop Policy Optimization in RL
url: https://www.emergentmind.com/topics/dual-loop-policy-optimization
type: topic
---

# Dual-Loop Policy Optimization in RL

Dual-Loop Policy Optimization denotes a class of reinforcement-learning and agent-learning procedures in which two coupled optimization loops are interleaved. In the most explicit recent usage, it names a metacognitive training scheme that “disentangles immediate decision-making from long-term capability growth”: the inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning from expert feedback [2603.07972]. Closely related structures appear earlier under primal-dual optimization for constrained MDPs [1802.06480], two-timescale projected gradient descent-ascent for regularized MDPs [2505.04494], dual formulations for policy constraints [2404.16468], referential and conservative updates in model-based RL [2209.07676], Dual Policy Iteration [1805.10755], bi-level offline pessimism [2310.06268], and primal-dual ergodic-risk constrained LQR [2503.05878]. The shared pattern is a decomposition into two interacting updates that operate on different objects, timescales, or objectives.

## 1. Scope and recurring structure

The cited literature uses a dual-loop design in several technically distinct ways. In constrained RL, the split is typically between a primal policy update and a dual multiplier update. In regularized LP formulations, it is a fast primal descent and a slow dual ascent. In multi-agent LLM systems, it is an RL loop for “when to ask for help” coupled with a supervised loop for “how to incorporate help.” In planner-learner systems, it is an alternating update between a slow expert policy and a fast reactive policy. In offline RL, it becomes a bi-level interaction between pessimistic value construction and policy improvement.

| Framework | First loop | Second loop |
|---|---|---|
| APDO [1802.06480] | On-policy primal policy update | Dual ascent with a one-time off-policy adjustment |
| PGDA-RL [2505.04494] | Fast update of \(V\) | Slow update of discounted occupancy \(\rho\) |
| HILA DLPO [2603.07972] | GRPO on deferral actions | Continual learning / SFT on expert demonstrations |
| DPI [1805.10755] | Non-reactive expert improvement | Reactive-policy imitation-style update |
| CDPO [2209.07676] | Referential update under a reference model | Conservative trust-region update |
| Bi-level offline RL [2310.06268] | Lower-level confidence-set construction | Upper-level conservative policy optimization |
| ER-LQR [2503.05878] | Inner policy optimization over \(K\) | Outer multiplier update over \(\lambda\) |

A plausible unifying interpretation is that “dual-loop” does not identify a single algorithmic template; it identifies a design principle for nested or alternating optimization. The loops may be primal/dual, planner/learner, RL/SFT, or pessimistic evaluator/policy improver.

## 2. Lagrangian primal-dual formulations

In constrained Markov Decision Processes, the canonical dual-loop construction starts from a constrained objective. One formulation maximizes the \(\gamma\)-discounted return
\[
R(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t R(s_t,a_t,s_{t+1})\Bigr]
\]
subject to long-term cost bounds
\[
C_i(\pi_\theta)=\mathbb E_{\tau\sim \pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t C_i(s_t,a_t,s_{t+1})\Bigr]\le d_i,
\]
with Lagrangian
\[
\mathcal L(\theta,\lambda)=R(\pi_\theta)-\sum_{i=1}^m \lambda_i\,[C_i(\pi_\theta)-d_i], \qquad \lambda\ge 0,
\]
and saddle-point objective \(\min_{\lambda\ge 0}\max_\theta \mathcal L(\theta,\lambda)\) [1802.06480]. The inner loop ascends in \(\theta\) using an on-policy likelihood-ratio gradient, often through a TRPO/PPO-style surrogate objective
\[
L_{\mathrm{surr}}(\theta;\theta_k,\lambda_k)=\mathbb E_t[\rho_t(\theta)\,\hat A_t^L],
\]
where \(\hat A_t^L\) is a Lagrangian advantage estimate [1802.06480].

A related generic constrained-RL formulation writes
\[
\max_\pi J(\pi)\quad \text{s.t.}\quad C_i(\pi)\le 0,
\]
with
\[
\mathcal L(\pi,\lambda)=J(\pi)+\sum_{i=1}^m \lambda_i\,C_i(\pi),
\]
and interleaved gradient-based updates
\[
\theta_{k+1}\leftarrow \theta_k+\alpha_p\nabla_\theta \mathcal L(\pi_{\theta_k},\lambda_k),\qquad
\lambda_i^{k+1}\leftarrow [\max(0,\lambda_i^k+\alpha_d\,C_i(\pi_{\theta_k}))].
\]
This same framework yields a direct reward-shaping interpretation through
\[
r'(s,a,s')=r(s,a,s')+\sum_i \lambda_i\,r_i(s,a,s'),
\]
so the dual variables act as trainable reward modifiers [2404.16468].

These formulations establish the most classical sense of dual-loop optimization in RL: the policy is optimized under a fixed multiplier, and the multiplier is updated to enforce feasibility. This suggests that the “dual” in dual-loop often refers literally to Lagrange duality, although later work broadens the pattern beyond multiplier updates.

## 3. Two-timescale separation and off-policy acceleration

The main algorithmic difficulty in primal-dual methods is that naïve alternation can be sample-inefficient or unstable. “Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning” addresses this in CMDPs by noting that existing methods only use on-policy data for dual updates, which results in sample inefficiency and slow convergence [1802.06480]. APDO keeps the primal update on-policy, but after \(K^{adj}\) iterations performs a one-time jump to an approximately optimal multiplier \(\lambda^{OFF}\) estimated offline from a replay buffer. Concretely, it stores every transition in a replay buffer \(\mathcal R\), runs an off-policy primal-dual algorithm on \(\mathcal R\) at \(k=K^{adj}\), resets \(\lambda^{K^{adj}+1}\leftarrow \lambda^{OFF}\), and then resumes on-policy dual ascent [1802.06480].

In the reported MuJoCo point-gather experiment, APDO and CPO both maintain long-term cost at the \(0.2\) limit, PDO violates the constraint for many epochs, and APDO reaches an average return of \(\approx 11\) in \(\sim 45\) epochs whereas CPO requires \(\sim 90\) epochs; the dual variable exhibits a pronounced jump at \(k=K^{adj}\) and then settles near the optimal multiplier \(\lambda^*\) [1802.06480]. The paper simultaneously states that it does not provide formal convergence proofs or sample-complexity bounds for APDO.

A more theoretically explicit two-timescale construction appears in PGDA-RL. There, a regularized LP reformulation of a finite MDP introduces a primal variable \(V\in \mathbb R^{|\mathcal S|}\) and a dual discounted occupancy \(\rho\in \mathbb R_+^{|\mathcal S|\times |\mathcal A|}\), with saddle objective
\[
L(V,\rho)=\frac{\eta_V}{2}\|V\|_2^2+\sum_{s,a}\rho(s,a)\Bigl[\Delta[V](s,a)-\eta_\rho\ln\frac{\rho(s,a)}{\tilde \rho(s)}\Bigr],
\]
where \(\tilde \rho(s)=\sum_a \rho(s,a)\) [2505.04494]. The key algorithmic choice is a two-timescale stochastic approximation regime: the “fast” variable \(V\) is updated with step size \(\alpha_k\), the “slow” variable \(\rho\) with \(\beta_k\), and \(\beta_k/\alpha_k\to 0\) [2505.04494]. In the online version, the algorithm operates from a single Markov trajectory, uses small replay buffers \(\mathcal D(s,a)\) and incoming lists \(\mathcal D_{inc}(s)\), updates only visited coordinates, and may choose actions according to
\[
\pi_k(a|s)=(1-\epsilon_k)\,\rho_k(s,a)/\tilde \rho_k(s)+\epsilon_k\,U(\mathcal A)
\]
to guarantee sufficient exploration [2505.04494].

Under finite-state assumptions, boundedness, Lipschitz gradients, ergodicity of the on-policy chain, strictly positive exploration, and diminishing step sizes satisfying
\[
\sum \alpha_k=\sum \beta_k=+\infty,\qquad \sum \alpha_k^2+\sum \beta_k^2<+\infty,\qquad \beta_k/\alpha_k\to 0,
\]
PGDA-RL proves almost-sure convergence
\[
(V_k,\rho_k)\xrightarrow{\mathrm{a.s.}}(V^*,\rho^*),
\]
and therefore \(\pi_k\to \pi^*\), without requiring a simulator or a fixed behavioral policy [2505.04494].

Taken together, these works show two distinct reasons for a dual-loop split: acceleration through replay-assisted outer updates, and asymptotic analysis through timescale separation.

## 4. Metacognitive Dual-Loop Policy Optimization for multi-agent LLMs

In “Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning,” Dual-Loop Policy Optimization is the explicit name of the training mechanism inside the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework [2603.07972]. The metacognitive decision problem is modeled as an MDP whose state \(s_t\) is a compact “cognitive” embedding of task context \(x_t^{task}\), self context \(x_t^{self}\), peer context \(x_t^{peer}\), and optional structured cues \(z_t=(z_t^{soc},z_t^{mon},z_t^{ctrl})\). The action space is
\[
A=\{a_{eval},a_{create},a_{defer}\},
\]
and the policy \(\pi_\theta(a|s)\) outputs a distribution over these actions [2603.07972].

The inner loop uses GRPO with a cost-aware reward
\[
R(s_t,a_t)=
\begin{cases}
R_{gt}(y(a_t)), & a_t=a_{eval},\\
R_{gt}(y(a_t))-C_{create}, & a_t=a_{create},\\
R_{gt}(y(a_t))-C_{defer}, & a_t=a_{defer},
\end{cases}
\]
where \(R_{gt}(y)\in\{0,1\}\), \(C_{create}\ge 0\), and \(C_{defer}>C_{create}\) [2603.07972]. At each state, with \(k=|A|\) candidate actions, the method forms a reward vector \(\mathbf R_t=[R(s_t,a_1),\dots,R(s_t,a_k)]\) and defines centered advantages
\[
A(s_t,a_i)=R(s_t,a_i)-\frac{1}{k}\sum_{j=1}^k R(s_t,a_j).
\]
The policy-gradient surrogate is
\[
\mathcal L_{PG}(\theta)=-\mathbb E_{s_t\sim\mathcal D,\,a_t\sim \pi_\theta}[A(s_t,a_t)\log \pi_\theta(a_t|s_t)],
\]
augmented by a KL penalty to a fixed reference policy \(\pi_{ref}\) and an entropy bonus. The full inner-loop loss is
\[
\mathcal L_{inner}(\theta)=\mathcal L_{PG}(\theta)+\mathcal L_{KL}(\theta)+\mathcal L_{ent}(\theta)
\]
[2603.07972].

The outer loop is continual learning. Whenever the policy selects \(a_{defer}\) and receives an expert demonstration \(y^{human}=(t_1,\dots,t_L)\), the system turns that demonstration into a supervised fine-tuning sample with loss
\[
\mathcal L_{SFT}(\theta)=-\sum_{i=1}^{L}\log \pi_\theta\bigl(t_i\mid s_t,t_{1:i-1}\bigr),
\]
and back-propagates it only on deferral steps [2603.07972]. The combined training schedule maintains replay buffers \(B_{RL}\) and \(B_{SFT}\), performs GRPO updates from \(B_{RL}\), performs SFT updates from \(B_{SFT}\), and defines the total loss
\[
\mathcal L_{total}(\theta)=\mathcal L_{inner}(\theta)+\alpha_{sft}\,\mathbb I_{\{a_t=\mathrm{defer}\}}\,\mathcal L_{SFT}(\theta)
\]
[2603.07972].

The paper’s claims about performance are explicitly empirical. It reports that, as training progresses from the random meta-policy to GRPO and then to full DLPO, the system moves toward higher task accuracy while simultaneously reducing the DEFER rate; GRPO alone rapidly learns to avoid overly frequent deferral but plateaus in accuracy, whereas adding the continual-learning loop yields further accuracy gains while deferral continues to shrink [2603.07972]. It also states that the inner-loop KL penalty and entropy bonus resemble TRPO/PPO-style stabilization, but it does not provide a formal theorem on convergence rates [2603.07972].

## 5. Planner-learner and conservative trust-region variants

A different lineage of dual-loop optimization does not center on dual variables at all. “Dual Policy Iteration” maintains two stationary policies at each iteration: a reactive policy \(\pi_n(a|s)\), typically a parametric function approximator deployed at test time, and a non-reactive “expert” policy \(\eta_n(a|s)\), typically a slower planner such as local LQR or tree search [1805.10755]. The overall viewpoint is
\[
\min_\pi \max_\eta \; \mathbb E_{s\sim d_\pi,\,a\sim \pi(\cdot|s)}[A^\eta(s,a)],
\]
whose unique Nash equilibrium is \((\pi,\eta)=(\pi^*,\pi^*)\) [1805.10755]. The two loops alternate as follows: first, fit a local model \(\hat P_n\) from rollout data; second, update \(\eta_n\) by model-based optimal control under a trust-region constraint around \(\pi_n\); third, update \(\pi_{n+1}\) by minimizing the expert’s local disadvantage under its own trust region [1805.10755]. The paper gives a convergence analysis extending existing approximate policy-iteration theory and notes that, when \(\alpha\to 0\), the method reduces to CPI/TRPO, while forward-search experts recover ExIt or AlphaGo-Zero-style schemes [1805.10755].

“Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning” uses yet another two-stage construction [2209.07676]. At each iteration, it first performs a Referential Update under a reference model \(\tilde f_t\in F_t\), for instance the least-squares estimate \(\hat f_t^{LS}\), by setting
\[
q_t=\arg\max_q V_q^{\tilde f_t}.
\]
It then performs a Conservative Update
\[
\pi_t=\arg\max_\pi \mathbb E_{f\sim posterior}[V_\pi^f]
\quad\text{s.t.}\quad
\mathbb E_{s\sim \nu_{q_t}}[D_{TV}(\pi(\cdot|s),q_t(\cdot|s))]\le \eta,
\]
or, in practice, a KL-based variant via Pinsker’s inequality [2209.07676]. This split is dual-loop in the sense of first producing a stable reference-improving policy \(q_t\), then performing a conservative posterior-expectation update to \(\pi_t\).

The theoretical profile of CDPO is unusually explicit. If a posterior-sampling algorithm satisfies \(BayesRegret(T,\pi^{PSRL})\le D\), then CDPO satisfies
\[
BayesRegret(T,\pi^{CDPO})\le 3D.
\]
Under the stated regularity conditions it also establishes a monotonic policy-improvement result and a global-optimality result
\[
BayesRegret(T)=\tilde O(L\,\sigma \sqrt{d_K\,d_E\,T}),
\]
matching the \(\tilde O(\sqrt T)\) rate of PSRL/OFU up to complexity factors [2209.07676]. The cited discussion contrasts this with OFU and PSRL: OFU can over-explore when the confidence set is large, while PSRL can make aggressive jumps when sampled models fluctuate; CDPO uses a stable reference and an expectation-based conservative update to mitigate those effects [2209.07676].

These planner-learner and trust-region variants broaden the meaning of dual-loop optimization. Here the two loops are not primal and dual variables; they are two policy-improvement operators with different stability and exploration roles.

## 6. Bi-level pessimism, ergodic-risk control, and theoretical limits

In offline RL, a dual-loop or bi-level structure appears as a hierarchy between policy improvement and pessimistic value construction. “Bi-Level Offline Policy Optimization with Limited Exploration” first builds, for a fixed policy \(\pi\), a confidence set of \(q\)-functions whose weighted Bellman errors are small:
\[
\mathcal Q_{\varepsilon_n}
=
\Bigl\{
q\in \mathcal Q:
\sup_{\tau\in \widetilde \Omega_{\widetilde \sigma_n}}
|\widehat M_n(q,\tau)|
\le \varepsilon_n
\Bigr\},
\]
where \(\widetilde \Omega_{\widetilde \sigma_n}\) constrains a nonnegative weighting function \(\tau\) through a detection penalty \(\xi_n(\mathbb D,\tau)\) [2310.06268]. The upper level then chooses
\[
\widehat \pi=\arg\max_{\pi\in \Pi}\underline q^\pi(s^0,\pi),
\]
where \(\underline q^\pi\) is the most pessimistic value estimate in the confidence set [2310.06268]. The practical solver introduces a penalized saddle-point loss
\[
\mathcal L(q,\tau,\pi;c^*,\lambda)
=
q(s^0,\pi)
+
\frac{1}{n(1-\gamma)}
\Bigl\{
c^*\Bigl|\sum_i\tau(s_i,a_i)\bigl[q(s_i,a_i)-r_i-\gamma q(s_i',\pi)\bigr]\Bigr|
-\lambda\sum_i \mathbb D(\tau(s_i,a_i))
\Bigr\},
\]
alternates proximal updates on \(\tau\) and closed-form updates on \(q\), and then updates the policy by a mirror-descent step with negative-entropy divergence [2310.06268]. Its theoretical guarantee is explicitly “best-effort”: under realizability and boundedness, and without coverage or completeness assumptions, the learned policy satisfies the stated regret bound against any comparator policy [2310.06268].

A more classical control-theoretic dual-loop appears in ergodic-risk constrained LQR. There the system is
\[
x_{t+1}=Ax_t+Bu_t+Hw_{t+1},
\]
with stationary linear feedback \(u_t=Kx_t\), average cost
\[
J(K)=\mathrm{Tr}[(Q+K^\top R K)\Sigma_K],
\]
and ergodic-risk constraint
\[
\gamma_N^2(K)\le \bar \beta
\]
based on the asymptotic conditional variance of the uncertainty increment \(C_t\) [2503.05878]. The constrained optimization problem is
\[
\min_{K\in S}J(K)\quad \text{s.t.}\quad \gamma_N^2(K)\le \bar \beta,
\]
with Lagrangian
\[
\mathcal L(K,\lambda)=J(K)+\lambda(\gamma_N^2(K)-\bar \beta),\qquad \lambda\ge 0.
\]
For fixed \(\lambda\), the policy gradient has closed form:
\[
\nabla_K \mathcal L(K,\lambda)=2(RK+B^\top P_{K,\lambda}A_K)\Sigma_K
\]
[2503.05878]. The algorithm performs an inner loop over \(K\) until \(\|\nabla_K \mathcal L(K,\lambda_m)\|_F\le \epsilon_{inner}\), then an outer projected ascent
\[
\lambda_{m+1}=[\lambda_m+\alpha_{dual}(\gamma_N^2(K_{m+1})-\bar \beta)]_+.
\]
Under the stated assumptions, the inner loop converges quadratically via Riemannian Newton or linearly via gradient descent, the outer loop converges at rate \(O(1/\sqrt M)\) in the duality gap, and the overall complexity is \(O(\log\log(1/\epsilon_{inner})/\epsilon_{outer}^2)\) [2503.05878]. In the reported Grumman X-29 experiment with Student-\(t\) process noise, ER-LQR achieves target risk \(\bar \beta=0.8\,\gamma_N^2(K_{LQR})\) while incurring only \(0.25\%\) increase in \(J\) [2503.05878].

The literature therefore places strong limits on any attempt to treat dual-loop policy optimization as a single, fully unified theory. Some instances provide almost-sure convergence to a unique regularized optimum [2505.04494]; some provide monotonic-improvement or regret guarantees [2209.07676, 1805.10755, 2503.05878, 2310.06268]; some explicitly report the absence of formal convergence proofs [1802.06480]; and some provide empirical rather than theorem-level evidence [2603.07972]. A common misconception is that dual-loop necessarily means a Lagrange-multiplier method. The cited work instead shows several non-equivalent meanings: multiplier enforcement in CMDPs, timescale separation in regularized LPs, planner-learner alternation in approximate policy iteration, conservative trust-region refinement in model-based RL, and RL/SFT coupling in human-in-the-loop multi-agent LLM systems.

Source: https://www.emergentmind.com/topics/dual-loop-policy-optimization