---
title: Double-Loop Multi-Agent (DLMA) Framework
url: https://www.emergentmind.com/topics/double-loop-multi-agent-dlma
type: topic
---

# Double-Loop Multi-Agent (DLMA) Framework

Searching arXiv for the cited papers to ground the article in the current record.
Double-Loop Multi-Agent (DLMA) denotes a class of multi-agent LLM systems that separate a fast operational loop from a slower loop that revises the conditions under which operation occurs. In the most explicit formulation, DLMA is a framework for automated scientific research in which a **leader loop** evolves research plans and a **follower loop** executes and adapts those plans [2510.06761]. Closely related formulations appear in Human-In-the-Loop Multi-Agent Collaboration (HILA) with Dual-Loop Policy Optimization, where an **inner loop** learns a metacognitive deferral policy and an **outer loop** continually updates the underlying models from expert feedback [2603.07972], and in RecursiveMAS, where an **inner operative process** performs latent recursive multi-agent computation while an **outer learning loop** co-optimizes cross-agent recursive links across recursion rounds [2604.25917]. Taken together, these works suggest that DLMA is best understood as an architectural pattern for bilevel or dual-timescale coordination rather than as a single fixed algorithm.

## 1. Conceptual definition

The clearest explicit definition of DLMA appears in the formulation of automated research as a bilevel optimization problem:

$$
\begin{aligned}
\max_{p \in \mathcal{P}} \quad & R(p, y^*(p)) \\
\text{s.t.} \quad & y^*(p) \in \arg\max_{y \in \mathcal{Y}(p)} f(p, y).
\end{aligned}
$$

Here, the upper level selects a research plan \(p\), and the lower level selects an execution sequence \(y\) conditioned on that plan [2510.06761]. The same paper contrasts **single loop** and **double loop** directly: a single loop adjusts actions given fixed goals, whereas a double loop also questions and changes the goals themselves. This distinction supplies the conceptual core of DLMA.

In HILA, the dual-loop distinction is expressed differently. The **inner loop** performs reinforcement-learning-based optimization of a metacognitive policy over the action set \(\{a_{\text{eval}}, a_{\text{create}}, a_{\text{defer}}\}\), while the **outer loop** converts expert feedback collected through \(a_{\text{defer}}\) into supervised fine-tuning data that expands the base model’s capabilities [2603.07972]. In RecursiveMAS, the terminology shifts again: the system executes recursive latent-space computation within and across agents, and training proceeds through an **inner–outer loop learning algorithm** for whole-system co-optimization through shared gradient-based credit assignment across recursion rounds [2604.25917].

A common misconception is to equate DLMA with any multi-round or multi-agent prompting scheme. The cited formulations are narrower. They require an explicit decomposition between a loop that handles immediate coordination or execution and another loop that changes plans, policies, or collaboration operators.

## 2. Formal structures and loop semantics

The three formulations instantiate the double-loop idea with different mathematical objects. In DLMA for automated research, the upper loop operates over proposal populations \(\mathcal{T}^{(g)} = \{ t_1^{(g)}, \ldots, t_N^{(g)} \}\), and the lower loop operates over an evolving to-do list \(P^{(0)} = \{P_1^{(0)}, \ldots, P_S^{(0)}\}\) whose steps are revised as drafting and experimentation proceed [2510.06761]. The leader loop therefore changes the plan variable itself, while the follower loop changes stepwise realizations of that plan.

In HILA, the inner loop is formalized as a Meta-MDP \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)\) over a structured metacognitive state, high-level actions, cost-aware rewards, and multi-round transitions. The reward is

$$
R(s_t, a_t) =
\begin{cases}
R_{\text{gt}}(y(a_t)), & a_t = \text{EVAL},\\
R_{\text{gt}}(y(a_t)) - C_{\text{create}}, & a_t = \text{CREATE},\\
R_{\text{gt}}(y_{\text{human}}(a_t)) - C_{\text{defer}}, & a_t = \text{DEFER}.
\end{cases}
$$

The outer loop is expressed through supervised fine-tuning on expert demonstrations,

$$
\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{i=1}^L \log \pi_\theta(t_i \mid s_t, t_{1:i-1}),
$$

and the joint dual-loop objective is

$$
\mathcal{L}_{\text{total}}(\theta) =
\mathbb{E}_{(s_t, a_t)}
\big[
\mathcal{L}_{\text{Inner}}(\theta)
+
\lambda_{\text{SFT}}
\cdot \mathbf{I}(a_t = a_{\text{defer}})
\cdot \mathcal{L}_{\text{SFT}}(\theta)
\big].
$$

This makes the two loops operationally distinct: the inner loop optimizes *when* to ask, and the outer loop improves *what* can be done autonomously [2603.07972].

In RecursiveMAS, the formal object is a recursive system state \(\mathcal{H} = \{H_1,\dots,H_N\}\), refined across recursion rounds as a unified latent-space computation. The system recurrence is written as

$$
\mathcal{H}^{(r)} = F_\phi(\mathcal{H}^{(r-1)}, x),
$$

and the outer loss is the cross-entropy over the final decoded output after unrolling the full recursion:

$$
\mathcal{L}_{\mathrm{out}}
= \mathrm{CE}\!\left(
\mathcal{S}^{(n)}\!\bigl(
\mathcal{S}^{(n-1)}(
\cdots \mathcal{S}^{(1)}(x)
)
\bigr),
y
\right).
$$

This suggests a DLMA semantics in which the inner loop is the latent recursive computation itself and the outer loop is gradient-based shaping of that recursion [2604.25917].

## 3. Architectural patterns

A concise comparison of the reported DLMA-style instantiations is useful.

| System | Inner loop | Outer loop |
|---|---|---|
| DLMA for automated research | Doctoral student agents execute and adapt a chosen plan | Professor agents evolve research plans via evolutionary meetings |
| HILA + DLPO | GRPO-trained metacognitive policy over EVAL, CREATE, DEFER | Continual supervised fine-tuning from expert demonstrations |
| RecursiveMAS | Latent autoregression within agents and latent transfer across agents | Co-optimization of RecursiveLinks across recursion rounds |

In the explicit DLMA research framework, the architecture is role-specialized. **Professor agents** generate, critique, and combine research proposals; **doctoral student agents** implement plans through drafting, coding, experiments, and replanning [2510.06761]. The leader loop uses three meeting operators—**involvement meeting** \(\phi(\cdot)\), **improvement meeting** \(\sigma(\cdot)\), and **integration meeting** \(\delta(\cdot)\)—followed by an LLM-based review panel and top-\(K\) selection.

HILA instead organizes a set of \(N\) autonomous LLM agents acting in rounds on a shared metacognitive state containing task context, self context, peer context, and optional structured signals \(z^{\text{soc}}, z^{\text{mon}}, z^{\text{ctrl}}\) [2603.07972]. There is no hard-coded role specialization, but CREATE, EVAL, and DEFER induce an emergent division of labor. CREATE acts as proposal generation, EVAL as consensus building, and DEFER as escalation.

RecursiveMAS replaces text-level coordination with latent-space communication. Agents are linked by **RecursiveLink** modules, with an inner form

$$
\mathcal{R}_\text{in}(h) = h + W_2\,\sigma(W_1 h),
$$

and an outer form

$$
\mathcal{R}_\text{out}(h) = W_3 h + W_2\,\sigma(W_1 h).
$$

The reported collaboration patterns are **Sequential Style (Planner–Critic–Solver)**, **Mixture Style (Parallel Specialists + Summarizer)**, **Distillation Style (Expert–Learner)**, and **Deliberation Style (Reflector–Tool-Caller)** [2604.25917]. A plausible implication is that DLMA is compatible with both explicit role hierarchies and more implicit operator-based coordination.

## 4. Optimization mechanisms

The optimization logic of DLMA varies substantially across instantiations, but each case preserves the separation between operational control and slower adaptation.

In HILA, the inner loop uses **Group Relative Policy Optimization (GRPO)**. Advantages are computed by centering rewards across a group of candidate actions:

$$
A(s_t, a_k) = R(s_t, a_k) - \frac{1}{K} \sum_{j=1}^K R(s_t, a_j).
$$

The surrogate policy-gradient loss is

$$
\mathcal{L}_{\text{PG}}(\theta)
=
- \mathbb{E}_{s_t, a_t \sim \pi_\theta}
\big[
A(s_t, a_t)\log \pi_\theta(a_t \mid s_t)
\big],
$$

and the inner-loop objective augments it with a KL penalty and entropy bonus:

$$
\mathcal{L}_{\text{Inner}}
=
\mathcal{L}_{\text{PG}}
+
\beta_{\text{KL}}\mathcal{L}_{\text{KL}}
-
\beta_{\text{ent}}\mathcal{L}_{\text{Entropy}}.
$$

This mechanism directly optimizes the trade-off between autonomous action and human deferral under a scalar cost model [2603.07972].

RecursiveMAS adopts a two-stage training schedule. First, each agent’s inner RecursiveLink is warm-started with the cosine-similarity latent-alignment objective

$$
\mathcal{L}_\mathrm{in}
=
1 - \cos\!\Bigl(
\mathcal{R}_\text{in}(H),
\mathrm{Emb}_{\theta_i}(y)
\Bigr),
$$

with the base LLM frozen. Second, the system is unrolled across recursion rounds and optimized with a system-level cross-entropy objective while updating outer RecursiveLinks [2604.25917]. The authors characterize this as “shared gradient-based credit assignment across recursion rounds.”

In the explicit automated-research DLMA, the leader loop resembles an evolutionary search procedure, while the follower loop resembles online replanning under contextual and external observations. The stepwise update rule for the current task is

$$
P_s^{(t+1)} = \pi(P_s^{(t)} \mid C_s, E_s),
$$

and future steps are revised post hoc as new drafts and execution evidence accumulate [2510.06761]. This suggests that DLMA need not be tied to RL or differentiable recursion; it can also be instantiated as discrete search plus adaptive execution.

## 5. Empirical behavior

The available empirical evidence presents DLMA as both a performance pattern and an efficiency or robustness pattern, depending on the instantiation.

For HILA on the LLaMA3-8B backbone, Table 1 reports **HILA (with DLPO)** at GSM8K **89.86**, AMC **35.83**, AIME **9.37**, HumanEval **72.15**, and MMLU **73.62**, compared with **Vanilla SA** at GSM8K **72.76**, AMC **8.03**, AIME **2.96**, HumanEval **47.56**, and MMLU **57.99** [2603.07972]. Table 3 further separates stages: **HILA (Init Policy)** gives GSM8K **88.15**, AMC **33.33**, MMLU **68.30**; **HILA + GRPO** gives GSM8K **88.38**, AMC **32.50**, MMLU **70.47**; and **HILA + DLPO** gives GSM8K **89.86**, AMC **35.83**, MMLU **73.62**. The paper also reports a pronounced action-distribution shift on MMLU from **EVAL 47%, CREATE 34%, DEFER 19%** at initialization to **EVAL 74%, CREATE 21%, DEFER 5%** after DLPO. This indicates that the outer loop changes the competence profile of the agents, not merely the policy over actions.

For RecursiveMAS, the reported headline result is an average accuracy improvement of **8.3%**, together with **1.2\(\times\)-2.4\(\times\)** end-to-end inference speedup and **34.6%-75.6%** token usage reduction [2604.25917]. The same paper gives task-level examples: **MATH500 88.0 vs best baseline 85.8**, **AIME2025 86.7 vs best baseline 73.3**, **AIME2026 86.7 vs best baseline 76.7**, **GPQA-D 66.2 vs best baseline 62.8**, **LiveCodeBench 42.9 vs best baseline 39.8**, and **MedQA 79.3 vs best baseline 77.2**. Its complexity analysis attributes the efficiency gains to replacing per-step softmax projection with MLP-based latent transformation.

For the explicit DLMA automated-research system, ACLAward results report **Soundness 4.03**, **Excitement 4.02**, **Overall 4.01**, and **Confidence 4.13**, exceeding the reported baselines, including **Agent Laboratory** at **3.94 overall** and **Dolphin** at **3.93 overall** [2510.06761]. On Laboratory, DLMA reports **Soundness 3.93**, **Excitement 3.68**, **Overall 3.69**, and **Confidence 4.68**. The generational analysis on the Plagiarism dataset shows ACL mean scores rising from **3.86** at generation 1 to **3.99** at generation 5, while NeurIPS mean scores rise from **4.04** to **4.15**. The follower-loop support-rate analysis is presented as evidence that post-hoc replanning maintains high support across later steps where initial support would otherwise degrade.

## 6. Limitations, misconceptions, and open directions

The three instantiations expose different failure modes. In the automated-research DLMA, the authors identify **computational expense**, **code hallucinations**, **plateau in plan quality**, and **LLM evaluation bias**; full DLMA is reported at approximately **1,558 seconds** and **1,751,901** tokens per paper in Table 7 [2510.06761]. In HILA, the main assumptions include the use of **GPT-4o / GPT-4o-mini as human proxies**, a scalar deferral cost \(C_{\text{defer}}\), and a small discrete action set without complex hierarchical control [2603.07972]. In RecursiveMAS, recursion depth is tested only up to \(r=3\), collaboration topologies are pre-designed rather than learned, latent thoughts are not human-readable, and tool use is limited to a simple setting [2604.25917].

Another misconception is that all DLMA systems require humans in the loop. HILA does, explicitly, because DEFER queries a human expert and the outer loop learns from those demonstrations. RecursiveMAS does not; its outer loop is gradient-based optimization of RecursiveLinks. The automated-research DLMA occupies an intermediate position: it is multi-agent and tool-using, but its double-loop separation is between plan evolution and plan execution rather than between autonomy and human escalation.

Across the cited work, the open directions are consistent. The reported future directions include more dynamic role assignment and graph topologies, richer human participation modes, stronger evolutionary capabilities for MAS, improved paper–code alignment, richer evaluation protocols, and extensions beyond current LLM-only settings [2603.07972][2604.25917][2510.06761]. This suggests that DLMA is less a closed blueprint than a reusable systems principle: a multi-agent architecture in which one loop governs immediate behavior and another loop changes the agents, the plans, or the collaboration substrate itself.

Source: https://www.emergentmind.com/topics/double-loop-multi-agent-dlma