Double-Loop Multi-Agent (DLMA) Framework

• Double-Loop Multi-Agent (DLMA) is a framework that employs a dual-loop design, where the outer loop evolves research plans while the inner loop executes and adapts them.
• It integrates methods from evolutionary search, reinforcement learning, and gradient-based optimization to achieve bilevel coordination and role specialization.
• Empirical studies show that DLMA improves task performance and efficiency, though challenges remain regarding computation cost and dynamic role assignment.

Searching arXiv for the cited papers to ground the article in the current record. Double-Loop Multi-Agent (DLMA) denotes a class of multi-agent LLM systems that separate a fast operational loop from a slower loop that revises the conditions under which operation occurs. In the most explicit formulation, DLMA is a framework for automated scientific research in which a leader loop evolves research plans and a follower loop executes and adapts those plans (Zhang et al., 8 Oct 2025). Closely related formulations appear in Human-In-the-Loop Multi-Agent Collaboration (HILA) with Dual-Loop Policy Optimization, where an inner loop learns a metacognitive deferral policy and an outer loop continually updates the underlying models from expert feedback (Yang et al., 9 Mar 2026), and in RecursiveMAS, where an inner operative process performs latent recursive multi-agent computation while an outer learning loop co-optimizes cross-agent recursive links across recursion rounds (Yang et al., 28 Apr 2026). Taken together, these works suggest that DLMA is best understood as an architectural pattern for bilevel or dual-timescale coordination rather than as a single fixed algorithm.

1. Conceptual definition

The clearest explicit definition of DLMA appears in the formulation of automated research as a bilevel optimization problem:

$$\begin{aligned} \max_{p \in \mathcal{P}} \quad & R(p, y^*(p)) \ \text{s.t.} \quad & y^*(p) \in \arg\max_{y \in \mathcal{Y}(p)} f(p, y). \end{aligned}$$

Here, the upper level selects a research plan $p$, and the lower level selects an execution sequence $y$ conditioned on that plan (Zhang et al., 8 Oct 2025). The same paper contrasts single loop and double loop directly: a single loop adjusts actions given fixed goals, whereas a double loop also questions and changes the goals themselves. This distinction supplies the conceptual core of DLMA.

In HILA, the dual-loop distinction is expressed differently. The inner loop performs reinforcement-learning-based optimization of a metacognitive policy over the action set $$\{a_{\text{eval}}, a_{\text{create}}, a_{\text{defer}}\}$$, while the outer loop converts expert feedback collected through $a_{\text{defer}}$ into supervised fine-tuning data that expands the base model’s capabilities (Yang et al., 9 Mar 2026). In RecursiveMAS, the terminology shifts again: the system executes recursive latent-space computation within and across agents, and training proceeds through an inner–outer loop learning algorithm for whole-system co-optimization through shared gradient-based credit assignment across recursion rounds (Yang et al., 28 Apr 2026).

A common misconception is to equate DLMA with any multi-round or multi-agent prompting scheme. The cited formulations are narrower. They require an explicit decomposition between a loop that handles immediate coordination or execution and another loop that changes plans, policies, or collaboration operators.

2. Formal structures and loop semantics

The three formulations instantiate the double-loop idea with different mathematical objects. In DLMA for automated research, the upper loop operates over proposal populations $$\mathcal{T}^{(g)} = \{ t_1^{(g)}, \ldots, t_N^{(g)} \}$$, and the lower loop operates over an evolving to-do list $P^{(0)} = \{P_1^{(0)}, \ldots, P_S^{(0)}\}$ whose steps are revised as drafting and experimentation proceed (Zhang et al., 8 Oct 2025). The leader loop therefore changes the plan variable itself, while the follower loop changes stepwise realizations of that plan.

In HILA, the inner loop is formalized as a Meta-MDP $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)$$ over a structured metacognitive state, high-level actions, cost-aware rewards, and multi-round transitions. The reward is

$$R(s_t, a_t) = \begin{cases} R_{\text{gt}}(y(a_t)), & a_t = \text{EVAL},\ R_{\text{gt}}(y(a_t)) - C_{\text{create}}, & a_t = \text{CREATE},\ R_{\text{gt}}(y_{\text{human}}(a_t)) - C_{\text{defer}}, & a_t = \text{DEFER}. \end{cases}$$

The outer loop is expressed through supervised fine-tuning on expert demonstrations,

$$\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{i=1}^L \log \pi_\theta(t_i \mid s_t, t_{1:i-1}),$$

and the joint dual-loop objective is

$p$0

This makes the two loops operationally distinct: the inner loop optimizes when to ask, and the outer loop improves what can be done autonomously (Yang et al., 9 Mar 2026).

In RecursiveMAS, the formal object is a recursive system state $p$1, refined across recursion rounds as a unified latent-space computation. The system recurrence is written as

$p$2

and the outer loss is the cross-entropy over the final decoded output after unrolling the full recursion:

$p$3

This suggests a DLMA semantics in which the inner loop is the latent recursive computation itself and the outer loop is gradient-based shaping of that recursion (Yang et al., 28 Apr 2026).

3. Architectural patterns

A concise comparison of the reported DLMA-style instantiations is useful.

System	Inner loop	Outer loop
DLMA for automated research	Doctoral student agents execute and adapt a chosen plan	Professor agents evolve research plans via evolutionary meetings
HILA + DLPO	GRPO-trained metacognitive policy over EVAL, CREATE, DEFER	Continual supervised fine-tuning from expert demonstrations
RecursiveMAS	Latent autoregression within agents and latent transfer across agents	Co-optimization of RecursiveLinks across recursion rounds

In the explicit DLMA research framework, the architecture is role-specialized. Professor agents generate, critique, and combine research proposals; doctoral student agents implement plans through drafting, coding, experiments, and replanning (Zhang et al., 8 Oct 2025). The leader loop uses three meeting operators—involvement meeting $p$4, improvement meeting $p$5, and integration meeting $p$6—followed by an LLM-based review panel and top-$p$7 selection.

HILA instead organizes a set of $p$8 autonomous LLM agents acting in rounds on a shared metacognitive state containing task context, self context, peer context, and optional structured signals $p$9 (Yang et al., 9 Mar 2026). There is no hard-coded role specialization, but CREATE, EVAL, and DEFER induce an emergent division of labor. CREATE acts as proposal generation, EVAL as consensus building, and DEFER as escalation.

RecursiveMAS replaces text-level coordination with latent-space communication. Agents are linked by RecursiveLink modules, with an inner form

$y$0

and an outer form

$y$1

The reported collaboration patterns are Sequential Style (Planner–Critic–Solver), Mixture Style (Parallel Specialists + Summarizer), Distillation Style (Expert–Learner), and Deliberation Style (Reflector–Tool-Caller) (Yang et al., 28 Apr 2026). A plausible implication is that DLMA is compatible with both explicit role hierarchies and more implicit operator-based coordination.

4. Optimization mechanisms

The optimization logic of DLMA varies substantially across instantiations, but each case preserves the separation between operational control and slower adaptation.

In HILA, the inner loop uses Group Relative Policy Optimization (GRPO). Advantages are computed by centering rewards across a group of candidate actions:

$y$2

The surrogate policy-gradient loss is

$y$3

and the inner-loop objective augments it with a KL penalty and entropy bonus:

$y$4

This mechanism directly optimizes the trade-off between autonomous action and human deferral under a scalar cost model (Yang et al., 9 Mar 2026).

RecursiveMAS adopts a two-stage training schedule. First, each agent’s inner RecursiveLink is warm-started with the cosine-similarity latent-alignment objective

$y$5

with the base LLM frozen. Second, the system is unrolled across recursion rounds and optimized with a system-level cross-entropy objective while updating outer RecursiveLinks (Yang et al., 28 Apr 2026). The authors characterize this as “shared gradient-based credit assignment across recursion rounds.”

In the explicit automated-research DLMA, the leader loop resembles an evolutionary search procedure, while the follower loop resembles online replanning under contextual and external observations. The stepwise update rule for the current task is

$y$6

and future steps are revised post hoc as new drafts and execution evidence accumulate (Zhang et al., 8 Oct 2025). This suggests that DLMA need not be tied to RL or differentiable recursion; it can also be instantiated as discrete search plus adaptive execution.

5. Empirical behavior

The available empirical evidence presents DLMA as both a performance pattern and an efficiency or robustness pattern, depending on the instantiation.

For HILA on the LLaMA3-8B backbone, Table 1 reports HILA (with DLPO) at GSM8K 89.86, AMC 35.83, AIME 9.37, HumanEval 72.15, and MMLU 73.62, compared with Vanilla SA at GSM8K 72.76, AMC 8.03, AIME 2.96, HumanEval 47.56, and MMLU 57.99 (Yang et al., 9 Mar 2026). Table 3 further separates stages: HILA (Init Policy) gives GSM8K 88.15, AMC 33.33, MMLU 68.30; HILA + GRPO gives GSM8K 88.38, AMC 32.50, MMLU 70.47; and HILA + DLPO gives GSM8K 89.86, AMC 35.83, MMLU 73.62. The paper also reports a pronounced action-distribution shift on MMLU from EVAL 47%, CREATE 34%, DEFER 19% at initialization to EVAL 74%, CREATE 21%, DEFER 5% after DLPO. This indicates that the outer loop changes the competence profile of the agents, not merely the policy over actions.

For RecursiveMAS, the reported headline result is an average accuracy improvement of 8.3%, together with 1.2$y$7-2.4$y$8 end-to-end inference speedup and 34.6%-75.6% token usage reduction (Yang et al., 28 Apr 2026). The same paper gives task-level examples: MATH500 88.0 vs best baseline 85.8, AIME2025 86.7 vs best baseline 73.3, AIME2026 86.7 vs best baseline 76.7, GPQA-D 66.2 vs best baseline 62.8, LiveCodeBench 42.9 vs best baseline 39.8, and MedQA 79.3 vs best baseline 77.2. Its complexity analysis attributes the efficiency gains to replacing per-step softmax projection with MLP-based latent transformation.

For the explicit DLMA automated-research system, ACLAward results report Soundness 4.03, Excitement 4.02, Overall 4.01, and Confidence 4.13, exceeding the reported baselines, including Agent Laboratory at 3.94 overall and Dolphin at 3.93 overall (Zhang et al., 8 Oct 2025). On Laboratory, DLMA reports Soundness 3.93, Excitement 3.68, Overall 3.69, and Confidence 4.68. The generational analysis on the Plagiarism dataset shows ACL mean scores rising from 3.86 at generation 1 to 3.99 at generation 5, while NeurIPS mean scores rise from 4.04 to 4.15. The follower-loop support-rate analysis is presented as evidence that post-hoc replanning maintains high support across later steps where initial support would otherwise degrade.

6. Limitations, misconceptions, and open directions

The three instantiations expose different failure modes. In the automated-research DLMA, the authors identify computational expense, code hallucinations, plateau in plan quality, and LLM evaluation bias; full DLMA is reported at approximately 1,558 seconds and 1,751,901 tokens per paper in Table 7 (Zhang et al., 8 Oct 2025). In HILA, the main assumptions include the use of GPT-4o / GPT-4o-mini as human proxies, a scalar deferral cost $y$9, and a small discrete action set without complex hierarchical control (Yang et al., 9 Mar 2026). In RecursiveMAS, recursion depth is tested only up to $$\{a_{\text{eval}}, a_{\text{create}}, a_{\text{defer}}\}$$0, collaboration topologies are pre-designed rather than learned, latent thoughts are not human-readable, and tool use is limited to a simple setting (Yang et al., 28 Apr 2026).

Another misconception is that all DLMA systems require humans in the loop. HILA does, explicitly, because DEFER queries a human expert and the outer loop learns from those demonstrations. RecursiveMAS does not; its outer loop is gradient-based optimization of RecursiveLinks. The automated-research DLMA occupies an intermediate position: it is multi-agent and tool-using, but its double-loop separation is between plan evolution and plan execution rather than between autonomy and human escalation.

Across the cited work, the open directions are consistent. The reported future directions include more dynamic role assignment and graph topologies, richer human participation modes, stronger evolutionary capabilities for MAS, improved paper–code alignment, richer evaluation protocols, and extensions beyond current LLM-only settings (Yang et al., 9 Mar 2026, Yang et al., 28 Apr 2026, Zhang et al., 8 Oct 2025). This suggests that DLMA is less a closed blueprint than a reusable systems principle: a multi-agent architecture in which one loop governs immediate behavior and another loop changes the agents, the plans, or the collaboration substrate itself.