MASFly: Adaptive LLM Multi-Agent Framework

• MASFly is a dynamic LLM-based multi-agent framework that tailors its agent roster and communication topology in real time to meet query-specific needs.
• It employs retrieval-augmented SOP instantiation and a supervisory Watcher with a Personalized Experience Pool to correct anomalies during execution.
• Empirical evaluations show MASFly achieves a 61.7% success rate on the TravelPlanner benchmark, highlighting its robust adaptability and state-of-the-art performance.

Searching arXiv for the specified paper and closely related MAS literature. MASFly is an LLM-based multi-agent framework for dynamic adaptation at test time. Introduced in "MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time" (Liu et al., 14 Feb 2026), it is designed to avoid both manual multi-agent designs and "one-size-fits-all" automation by combining retrieval-augmented SOP instantiation for query-specific system generation with experience-guided supervision during execution. The framework is motivated by biological analogies—phenotypic plasticity for dynamic system generation and homeostatic regulation for real-time correction—and is reported to achieve state-of-the-art performance, including a 61.7\% success rate on the TravelPlanner benchmark, while exhibiting strong task adaptability and robustness (Liu et al., 14 Feb 2026).

1. Conceptual basis and problem setting

MASFly addresses a deployment-time limitation in LLM-based multi-agent systems: many existing systems are fixed after design time, even when downstream queries differ substantially in decomposition, tooling requirements, or interaction structure (Liu et al., 14 Feb 2026). Its core premise is that adaptation should occur at two levels. The first is system generation, where the agent roster and communication topology are instantiated for a specific query. The second is system execution, where the running system is monitored and corrected when anomalies emerge.

The framework is explicitly inspired by biological systems. In the paper’s formulation, phenotypic plasticity corresponds to dynamic system generation via reusable patterns stored as SOPs, while homeostatic regulation corresponds to real-time supervision and correction through the Watcher agent and the Personalized Experience Pool (PEP) (Liu et al., 14 Feb 2026). This framing emphasizes adaptation as an operational property of the deployed system rather than a property obtained only through pretraining or fine-tuning.

A common misunderstanding is that MASFly is primarily a fine-tuning method. The reported design instead centers on test-time system assembly, test-time supervision, and post-task reflective updates, and the paper states that the framework achieves continuous improvement without any extra LLM fine-tuning (Liu et al., 14 Feb 2026).

2. Architectural organization

The architecture comprises a repository of prior collaboration patterns, a retrieval-and-instantiation module, an execution substrate, a supervisory agent, and a reflective update mechanism (Liu et al., 14 Feb 2026). Each component has a distinct role in transforming a query into a task-specific executable multi-agent system and then refining the underlying knowledge stores after the task.

Component	Function
SOP Repository	Stores past successful collaboration patterns as cases
SOP Agent	Retrieves and instantiates query-specific SOPs into an executable MAS
Execution Engine	Runs the MAS and logs inter-agent messages and tool calls
Watcher Agent	Monitors running agents, consults the PEP, and intervenes on anomalies
Reflective Distillation Loop	Distills successful SOPs back into the repository and updates the PEP after failures

The paper also specifies the end-to-end data flow. A user first issues a query $Q$, after which the LLM extracts a need analysis $N$. Retrieval-augmented SOP instantiation then selects the top-$K$ SOPs from the repository and prompts the LLM to generate an Operation Procedure $OP$, defined as an executable configuration containing agent roles + communication graph. The MAS executes $OP$ under Watcher monitoring. If the Watcher detects an anomaly, it either sends experience-augmented corrective feedback or replaces a faulty agent. After completion, the system reflects: on success, it distills $OP$ into a new SOP entry; on failure, it diagnoses the failure and updates the PEP with records of the form $<Q,F,P>$ (Liu et al., 14 Feb 2026).

This organization makes the framework neither a static agent society nor a purely reactive controller. A plausible implication is that MASFly should be understood as a memory-augmented adaptive orchestration layer over LLM agents, because both system generation and system repair are grounded in structured traces of prior success and failure.

3. SOP repository and retrieval-augmented instantiation

The SOP Repository stores cases of the form

$C_i=(Q_i,N_i,S_i),$

where $Q_i$ is the original user query, $N_i$ is the LLM-generated need analysis, and $N$0 is the Standard Operating Procedure (Liu et al., 14 Feb 2026). Each $N$1 contains two elements: Agent Specifications and a Communication Structure. The agent specification set is written as

$N$2

where the tuple denotes name, responsibilities, detailed prompt, allowed tools. The communication structure is represented as a directed graph + textual description of information flow (Liu et al., 14 Feb 2026).

Given a new query $N$3, MASFly computes a need representation

$N$4

and scores each repository case by

$N$5

where $N$6 is cosine similarity of embeddings (Liu et al., 14 Feb 2026). The system retrieves the top-$N$7 cases and prompts the LLM to generate

$N$8

The resulting $N$9 is a customized multi-agent configuration consisting of agent prompts + communication topology and is ready for execution (Liu et al., 14 Feb 2026).

This retrieval-and-instantiation procedure is the framework’s principal mechanism for test-time structural adaptation. The paper’s description that SOPs function as a "genetic toolkit" is significant because it makes explicit that stored procedures are not replayed verbatim; they are retrieved, ranked, and recombined to instantiate a query-specific collaboration pattern (Liu et al., 14 Feb 2026). The reported ability to generate system structures ranging from 1 to 6+ agents follows directly from this query-dependent recombination process (Liu et al., 14 Feb 2026).

4. Experience-guided supervision and intervention

MASFly’s execution-time adaptation is mediated by the Watcher Agent and the Personalized Experience Pool (PEP) (Liu et al., 14 Feb 2026). The PEP stores failure records of the form

$K$0

where $K$1 denotes the failure cause and $K$2 is a set of items

$K$3

According to the update rule, after a failed execution on training data, the LLM receives the full logs and evaluation result and writes a new PEP entry (Liu et al., 14 Feb 2026).

The Watcher periodically retrieves relevant PEP entries for the current query. Its monitoring schedule is defined as every $K$4 messages or $K$5 tool-steps, and it operates on two levels: Inter-Agent Communication, including adherence to the workflow in $K$6 and loop avoidance, and Agent–Environment Interaction, including correct tool usage and response to dynamic feedback (Liu et al., 14 Feb 2026). When the Watcher detects an anomaly such as agent stalling or a wrong API call, it chooses one of two interventions. It may issue Experience-Augmented Guidance, sending a corrective message derived from a retrieved improvement strategy. Alternatively, it may perform Dynamic Agent Replacement, killing the faulty agent and spawning a new one with the original role specification from $K$7 (Liu et al., 14 Feb 2026).

The intervention policy is parameterized by the supervision interval $K$8, tool-step limit $K$9, and maximum interventions $OP$0. The paper reports that, empirically, $OP$1 and $OP$2 balance robustness vs.\ overhead (Liu et al., 14 Feb 2026). This is an empirical operating point rather than a learned policy, and the paper explicitly notes that supervision timing relies on heuristic $OP$3 and that automatic anomaly detectors could further optimize interventions (Liu et al., 14 Feb 2026).

A common misconception is that the Watcher is simply an observer. In the reported design, it is an active supervisory mechanism with authority to alter execution by sending corrective feedback or replacing agents. This distinction is central to MASFly’s notion of adaptive execution.

5. Algorithmic workflow and computational profile

The paper presents MASFly as a two-phase algorithm followed by a reflective update stage (Liu et al., 14 Feb 2026). In the System Generation phase, the system computes the need analysis, scores repository cases, selects the top-$OP$4 SOPs, generates $OP$5, and constructs the MAS. In the Adaptive Execution phase, the Watcher is initialized with the PEP and monitors the agents during round-based execution, triggering guidance or replacement when anomalies are detected. In the Reflective Distillation phase, which is marked as train only, success causes $OP$6 to be distilled into a new SOP entry and failure causes diagnosis followed by a new PEP entry (Liu et al., 14 Feb 2026).

The paper also specifies a computational profile. Retrieval requires

$OP$7

similarity computations, which the authors describe as feasible for $OP$8. Execution incurs

$OP$9

per task, where $OP$0 is the number of agents and $OP$1 is the number of communication rounds, plus Watcher checks every $OP$2 steps. Reflection adds one LLM call per training instance (Liu et al., 14 Feb 2026). The repository is also noted to grow with tasks, though the reported current size is $OP$3 SOPs (Liu et al., 14 Feb 2026).

These complexity claims locate MASFly in a regime where dynamic orchestration is practical when repository size remains modest. A plausible implication is that the framework’s scalability challenge is less about execution alone than about maintaining retrieval quality and supervision efficiency as the repository and experience pool expand.

6. Empirical performance, robustness, and limitations

The empirical evaluation spans planning, question answering, and code generation (Liu et al., 14 Feb 2026). The benchmarks and metrics are reported as follows: TravelPlanner (two-stage & sole-planning) with delivery rate, commonsense pass, hard-constraint pass, final-plan success; GAIA (Level 1/2/3) with accuracy on general QA; and HumanEval, MBPP, MBPP Pro with Pass@1 (Liu et al., 14 Feb 2026).

On TravelPlanner (sole-planning), the paper reports the following final success rates (Liu et al., 14 Feb 2026):

Method	Final Pass Rate
AgentSquare	32.2\%
SwarmAgentic	32.2\%
MASFly	61.7\%

For cross-task performance (average), the reported results are (Liu et al., 14 Feb 2026):

Benchmark	AgentSquare	MASFly
GAIA	35.0	39.5
HumanEval	96.84	98.73
MBPP	92.08	93.26
MBPP Pro	83.92	85.56

The ablation study on TravelPlanner common & hard rates isolates the contribution of the system’s principal components (Liu et al., 14 Feb 2026). The reported values are: MASFly at 85.3\% common and 49.5\% hard; w/o SOP-RAG at 65.3\% and 19.8\%; w/o SOP-Inst at 80.8\% and 40.9\%; w/o Watcher at 83.9\% and 37.1\%; and w/o PEP at 84.4\% and 45.2\%. These results indicate that both retrieval-augmented system generation and execution-time supervision contribute materially, with especially large degradation when SOP-RAG is removed.

The robustness test on MBPP Pro attack is designed by forcing the coding agent to output empty functions. Under this perturbation, the paper states that MASFly maintains near-original Pass@1, whereas baselines drop by 30–50 points (Liu et al., 14 Feb 2026). This directly supports the paper’s claim that the Watcher+PEP mechanism improves resilience under anomalies or adversarial prompts.

The limitations reported by the authors are specific. Supervision timing relies on heuristic $OP$4, and automatic anomaly detectors could further optimize interventions. In addition, repository size grows with tasks, although the reported current size remains $OP$5 SOPs (Liu et al., 14 Feb 2026). These limitations constrain the interpretation of the reported state-of-the-art results: the framework demonstrates dynamic adaptability and robustness under the evaluated settings, but some of its operating assumptions remain heuristic rather than fully learned or formally optimized.