---
title: 'MASFly: Adaptive LLM Multi-Agent Framework'
url: https://www.emergentmind.com/topics/masfly
type: topic
---

# MASFly: Adaptive LLM Multi-Agent Framework

Searching arXiv for the specified paper and closely related MAS literature.
MASFly is an LLM-based multi-agent framework for **dynamic adaptation at test time**. Introduced in "MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time" [2602.13671], it is designed to avoid both manual multi-agent designs and "one-size-fits-all" automation by combining **retrieval-augmented SOP instantiation** for query-specific system generation with **experience-guided supervision** during execution. The framework is motivated by biological analogies—**phenotypic plasticity** for dynamic system generation and **homeostatic regulation** for real-time correction—and is reported to achieve state-of-the-art performance, including a **61.7\% success rate on the TravelPlanner benchmark**, while exhibiting strong task adaptability and robustness [2602.13671].

## 1. Conceptual basis and problem setting

MASFly addresses a deployment-time limitation in LLM-based multi-agent systems: many existing systems are fixed after design time, even when downstream queries differ substantially in decomposition, tooling requirements, or interaction structure [2602.13671]. Its core premise is that adaptation should occur at two levels. The first is **system generation**, where the agent roster and communication topology are instantiated for a specific query. The second is **system execution**, where the running system is monitored and corrected when anomalies emerge.

The framework is explicitly inspired by biological systems. In the paper’s formulation, **phenotypic plasticity** corresponds to dynamic system generation via reusable patterns stored as SOPs, while **homeostatic regulation** corresponds to real-time supervision and correction through the Watcher agent and the Personalized Experience Pool (PEP) [2602.13671]. This framing emphasizes adaptation as an operational property of the deployed system rather than a property obtained only through pretraining or fine-tuning.

A common misunderstanding is that MASFly is primarily a fine-tuning method. The reported design instead centers on **test-time system assembly**, **test-time supervision**, and **post-task reflective updates**, and the paper states that the framework achieves continuous improvement **without any extra LLM fine-tuning** [2602.13671].

## 2. Architectural organization

The architecture comprises a repository of prior collaboration patterns, a retrieval-and-instantiation module, an execution substrate, a supervisory agent, and a reflective update mechanism [2602.13671]. Each component has a distinct role in transforming a query into a task-specific executable multi-agent system and then refining the underlying knowledge stores after the task.

| Component | Function |
|---|---|
| SOP Repository | Stores past successful collaboration patterns as cases |
| SOP Agent | Retrieves and instantiates query-specific SOPs into an executable MAS |
| Execution Engine | Runs the MAS and logs inter-agent messages and tool calls |
| Watcher Agent | Monitors running agents, consults the PEP, and intervenes on anomalies |
| Reflective Distillation Loop | Distills successful SOPs back into the repository and updates the PEP after failures |

The paper also specifies the end-to-end data flow. A user first issues a query $Q$, after which the LLM extracts a **need analysis** $N$. Retrieval-augmented SOP instantiation then selects the top-$K$ SOPs from the repository and prompts the LLM to generate an **Operation Procedure** $OP$, defined as an executable configuration containing **agent roles + communication graph**. The MAS executes $OP$ under Watcher monitoring. If the Watcher detects an anomaly, it either sends **experience-augmented corrective feedback** or **replaces a faulty agent**. After completion, the system reflects: on success, it distills $OP$ into a new SOP entry; on failure, it diagnoses the failure and updates the PEP with records of the form $<Q,F,P>$ [2602.13671].

This organization makes the framework neither a static agent society nor a purely reactive controller. A plausible implication is that MASFly should be understood as a **memory-augmented adaptive orchestration layer** over LLM agents, because both system generation and system repair are grounded in structured traces of prior success and failure.

## 3. SOP repository and retrieval-augmented instantiation

The SOP Repository stores cases of the form
$$
C_i=(Q_i,N_i,S_i),
$$
where $Q_i$ is the original user query, $N_i$ is the LLM-generated need analysis, and $S_i$ is the **Standard Operating Procedure** [2602.13671]. Each $S_i$ contains two elements: **Agent Specifications** and a **Communication Structure**. The agent specification set is written as
$$
\{A_{i,j}=(E_{i,j},R_{i,j},I_{i,j},T_{i,j})\},
$$
where the tuple denotes **name, responsibilities, detailed prompt, allowed tools**. The communication structure is represented as a **directed graph + textual description of information flow** [2602.13671].

Given a new query $Q$, MASFly computes a need representation
$$
N=\mathrm{LLM\_analyze}(Q)
$$
and scores each repository case by
$$
s(Q,C_i)=\lambda\,\mathrm{Sim}(Q,Q_i)+(1-\lambda)\,\mathrm{Sim}(N,N_i), \quad \lambda\in[0,1],
$$
where $\mathrm{Sim}(\cdot,\cdot)$ is **cosine similarity of embeddings** [2602.13671]. The system retrieves the top-$K$ cases and prompts the LLM to generate
$$
OP=\mathrm{LLM}(Q,\;N,\;\{S_i\}_{i=1}^K).
$$
The resulting $OP$ is a **customized multi-agent configuration** consisting of **agent prompts + communication topology** and is ready for execution [2602.13671].

This retrieval-and-instantiation procedure is the framework’s principal mechanism for test-time structural adaptation. The paper’s description that SOPs function as a **"genetic toolkit"** is significant because it makes explicit that stored procedures are not replayed verbatim; they are retrieved, ranked, and recombined to instantiate a query-specific collaboration pattern [2602.13671]. The reported ability to generate system structures ranging from **1 to 6+ agents** follows directly from this query-dependent recombination process [2602.13671].

## 4. Experience-guided supervision and intervention

MASFly’s execution-time adaptation is mediated by the **Watcher Agent** and the **Personalized Experience Pool (PEP)** [2602.13671]. The PEP stores failure records of the form
$$
<Q_i,\,F_i,\,P_i>,
$$
where $F_i$ denotes the **failure cause** and $P_i=\{p_{i,1},\dots,p_{i,m}\}$ is a set of items
$$
p_{i,k}=(\text{ErrorAttribution},\;\text{ImprovementStrategy}).
$$
According to the update rule, after a failed execution on training data, the LLM receives the full logs and evaluation result and writes a new PEP entry [2602.13671].

The Watcher periodically retrieves relevant PEP entries for the current query. Its monitoring schedule is defined as **every $M$ messages or $L$ tool-steps**, and it operates on two levels: **Inter-Agent Communication**, including adherence to the workflow in $OP$ and loop avoidance, and **Agent–Environment Interaction**, including correct tool usage and response to dynamic feedback [2602.13671]. When the Watcher detects an anomaly such as **agent stalling** or a **wrong API call**, it chooses one of two interventions. It may issue **Experience-Augmented Guidance**, sending a corrective message derived from a retrieved improvement strategy. Alternatively, it may perform **Dynamic Agent Replacement**, killing the faulty agent and spawning a new one with the original role specification from $OP$ [2602.13671].

The intervention policy is parameterized by the **supervision interval $M$**, **tool-step limit $L$**, and **maximum interventions $U$**. The paper reports that, empirically, **$M\approx|\text{team}|/2$ and $L=5$ balance robustness vs.\ overhead** [2602.13671]. This is an empirical operating point rather than a learned policy, and the paper explicitly notes that supervision timing relies on heuristic $M,L$ and that automatic anomaly detectors could further optimize interventions [2602.13671].

A common misconception is that the Watcher is simply an observer. In the reported design, it is an active supervisory mechanism with authority to alter execution by sending corrective feedback or replacing agents. This distinction is central to MASFly’s notion of **adaptive execution**.

## 5. Algorithmic workflow and computational profile

The paper presents MASFly as a two-phase algorithm followed by a reflective update stage [2602.13671]. In the **System Generation** phase, the system computes the need analysis, scores repository cases, selects the top-$K$ SOPs, generates $OP$, and constructs the MAS. In the **Adaptive Execution** phase, the Watcher is initialized with the PEP and monitors the agents during round-based execution, triggering guidance or replacement when anomalies are detected. In the **Reflective Distillation** phase, which is marked as **train only**, success causes $OP$ to be distilled into a new SOP entry and failure causes diagnosis followed by a new PEP entry [2602.13671].

The paper also specifies a computational profile. Retrieval requires
$$
O(|\mathcal{R}|)
$$
similarity computations, which the authors describe as feasible for **$|\mathcal{R}|<100$**. Execution incurs
$$
O(A\times T)
$$
per task, where $A$ is the number of agents and $T$ is the number of communication rounds, plus Watcher checks every $M$ steps. Reflection adds **one LLM call per training instance** [2602.13671]. The repository is also noted to grow with tasks, though the reported current size is **$<100$ SOPs** [2602.13671].

These complexity claims locate MASFly in a regime where dynamic orchestration is practical when repository size remains modest. A plausible implication is that the framework’s scalability challenge is less about execution alone than about maintaining retrieval quality and supervision efficiency as the repository and experience pool expand.

## 6. Empirical performance, robustness, and limitations

The empirical evaluation spans planning, question answering, and code generation [2602.13671]. The benchmarks and metrics are reported as follows: **TravelPlanner (two-stage & sole-planning)** with **delivery rate, commonsense pass, hard-constraint pass, final-plan success**; **GAIA (Level 1/2/3)** with **accuracy on general QA**; and **HumanEval, MBPP, MBPP Pro** with **Pass@1** [2602.13671].

On **TravelPlanner (sole-planning)**, the paper reports the following **final success rates** [2602.13671]:

| Method | Final Pass Rate |
|---|---|
| AgentSquare | 32.2\% |
| SwarmAgentic | 32.2\% |
| MASFly | **61.7\%** |

For **cross-task performance (average)**, the reported results are [2602.13671]:

| Benchmark | AgentSquare | MASFly |
|---|---:|---:|
| GAIA | 35.0 | **39.5** |
| HumanEval | 96.84 | **98.73** |
| MBPP | 92.08 | **93.26** |
| MBPP Pro | 83.92 | **85.56** |

The ablation study on **TravelPlanner common & hard rates** isolates the contribution of the system’s principal components [2602.13671]. The reported values are: **MASFly** at **85.3\%** common and **49.5\%** hard; **w/o SOP-RAG** at **65.3\%** and **19.8\%**; **w/o SOP-Inst** at **80.8\%** and **40.9\%**; **w/o Watcher** at **83.9\%** and **37.1\%**; and **w/o PEP** at **84.4\%** and **45.2\%**. These results indicate that both retrieval-augmented system generation and execution-time supervision contribute materially, with especially large degradation when **SOP-RAG** is removed.

The robustness test on **MBPP Pro attack** is designed by forcing the coding agent to output empty functions. Under this perturbation, the paper states that **MASFly maintains near-original Pass@1, whereas baselines drop by 30–50 points** [2602.13671]. This directly supports the paper’s claim that the Watcher+PEP mechanism improves resilience under anomalies or adversarial prompts.

The limitations reported by the authors are specific. **Supervision timing relies on heuristic $M,L$**, and **automatic anomaly detectors could further optimize interventions**. In addition, **repository size grows with tasks**, although the reported current size remains **$<100$ SOPs** [2602.13671]. These limitations constrain the interpretation of the reported state-of-the-art results: the framework demonstrates dynamic adaptability and robustness under the evaluated settings, but some of its operating assumptions remain heuristic rather than fully learned or formally optimized.

Source: https://www.emergentmind.com/topics/masfly