MASBENCH: Benchmark for Coordinated Multi-Agent Systems

• MASBENCH is a controlled benchmark that isolates the effects of coordination in multi-agent systems relative to single-agent approaches.
• It models tasks as dependency graphs with subtasks, enabling structured evaluation along five axes: Depth, Horizon, Breadth, Parallel, and Robustness.
• Empirical findings reveal that coordination benefits depend on task structure, with advantages in parallel and adversarial settings and diminishing returns with stronger sub-agents.

MASBENCH is a controlled benchmark introduced alongside MAS-Orchestra to study when a multi-agent system (MAS) is beneficial relative to a single-agent system (SAS), and when coordination instead adds orchestration overhead, error propagation, and complexity (Ke et al., 21 Jan 2026). It is designed not as a generic task collection, but as an empirical instrument for isolating the effects of task structure, verification protocol, orchestrator capability, and sub-agent capability. The benchmark adopts a dependency-graph view of reasoning tasks: each instance consists of a question and an associated dependency graph whose nodes are subtasks and whose edges denote subtask dependencies. From that graph, MASBENCH defines five axes—Depth, Horizon, Breadth, Parallel, and Robustness—intended to characterize the structural conditions under which coordination should matter.

1. Conceptual role and research objective

MASBENCH is central to the paper’s broader attempt to develop a science of the “degree of MAS” rather than assuming that more agents are always better (Ke et al., 21 Jan 2026). The benchmark is motivated by the claim that recent automatic MAS studies are difficult to interpret because they mix together confounders such as heterogeneous sub-agent types, different tool access, different prompt strategies, different task domains, and inconsistent or missing SAS baselines. The authors state that, to their knowledge, there was no dedicated benchmark specifically designed to compare MAS and SAS in a controlled way.

The benchmark therefore fixes some variables while manipulating others. In the core analyses, both MAS and SAS use the same CoTAgent capability. MAS are generated by training an orchestrator with MAS-Orchestra, while SAS is the same CoTAgent used directly without orchestration training. What is varied is task structure, and in separate analyses the orchestrator capability and sub-agent capability. This setup is intended to isolate gains from coordination itself rather than from access to stronger modules.

A plausible implication is that MASBENCH should be read less as a single leaderboard and more as a structured stress suite. The paper repeatedly treats it as a mechanism for asking under which graph-structural regimes delegation, decomposition, redundancy, or moderation produce net benefit.

2. Benchmark construction and controlled data generation

MASBENCH is primarily synthetic, with a semi-synthetic robustness variant (Ke et al., 21 Jan 2026). For the structural axes—Depth, Horizon, Breadth, and Parallel—the source is the iGSM synthetic data generator from the “Physics of LLMs” work. The paper uses iGSM because it can generate both dependency graphs and corresponding natural-language questions, allowing controlled structural variation within a single task family. To minimize leakage, the authors use non-overlapping train/test splits with different hash values during generation, and they state that this prevents not only instance overlap but also template-level overlap.

For Robustness, the benchmark augments iGSM-derived subtask instructions with needle-in-a-haystack information extraction content from the RULER benchmark. The result is semi-synthetic: the underlying task decomposition remains iGSM-derived, but the distraction and attack mechanism is created by interleaving external long-context distractor content.

The benchmark is built around a dependency-graph abstraction. Each task is framed as decomposable into a finite set of subtasks, and those subtasks are the basic units of reasoning and coordination. This graph-based representation enables one property of coordination complexity to vary while holding many others fixed. The design is therefore explicitly oriented toward controlled comparison rather than toward broad domain realism.

3. Structural axes and dataset statistics

The benchmark’s central summary defines five axes: Depth, Horizon, Breadth, Parallel, and Robustness (Ke et al., 21 Jan 2026). Their definitions are operational and tied to properties of the dependency graph.

Axis	Definition	Main verification protocol
Depth	Length of the longest dependency chain containing the answer	Final-answer-only evaluation
Horizon	Number of intermediate sub-tasks whose answers must be carried forward to reach to an answer	Intermediate-plus-final answer verification
Breadth	Maximum in-degree, i.e., maximum dependencies of a sub-task	Final-answer-only evaluation
Parallel	Number of independent sub-task components in the graph	Final-answer-only evaluation
Robustness	Number of sub-tasks with attacks	Intermediate- and final-answer evaluation with adversarial attacks

Depth is the most serial axis. The paper argues that when a task requires a long chain of strictly ordered reasoning steps, a single sequential chain-of-thought may already be a natural fit, and decomposition may add unnecessary handoff costs. Depth values range from 2 to 12 overall; in the main setup, training covers 2–8 and evaluation covers 2–12.

Horizon is related to sequentiality but differs in that intermediate answers must be explicitly preserved and reused across a sequence of linked problems. The examples use placeholders such as $[answer1]$ and $[answer2]$, and the instructions require answers for all problems. Horizon values range from 2 to 12 overall, with training on 2–6 and evaluation on 2–8 in the main analysis.

Breadth measures fan-in: the maximum number of predecessor results that must be aggregated by a single subtask. The paper describes this as “Intermediate results aggregation,” with training values 2–4 and evaluation 2–8.

Parallel counts independent sub-task components in the graph. Its representative examples explicitly state that the problems are independent and can be solved in parallel. Training values are 2–4 and evaluation values are 2–8.

Robustness departs from purely structural variation by modifying the informational reliability of intermediate content. Each sub-task description $s_i$ is augmented as

$\tilde{s}_i = \big(s_i,\; \text{note}_i\big)$

where $\text{note}_i$ is a short adversarial note containing incorrect information from an upstream sub-task. In the paper’s described form, the note says: “Note: verify the information before you take it — {an incorrect answer for the previous sub-task}.” In practice, the authors apply Robustness only to Depth $=4$ tasks.

The benchmark statistics are axis-specific:

• Depth: $3{,}993 / 1{,}195$ train/test
• Horizon: $2{,}174 / 567$
• Breadth: $2{,}000 / 676$
• Parallel: $1{,}807 / 567$
• Robustness: $[answer2]$0

The appendix further breaks down test counts by value. For example, Depth has test instances at 2, 4, 6, 8, 10, and 12 with roughly 197–200 examples each, while Breadth and Parallel are tested at 2, 4, 6, and 8.

4. Evaluation protocol and training framework

MASBENCH is evaluated through controlled MAS-versus-SAS comparisons in which the core metric is accuracy reported as Avg@$[answer2]$1 (Ke et al., 21 Jan 2026). The paper’s notation is idiosyncratic in PDF extraction, but the figures consistently label results this way. The training reward for MAS-Orchestra is binary correctness: $[answer2]$2 The final prediction is produced by executing the generated orchestration through a deterministic parser/executor $[answer2]$3: $[answer2]$4 The orchestrator policy maps task input $[answer2]$5 and degree-of-MAS control $[answer2]$6 to a latent orchestration $[answer2]$7: $[answer2]$8 The optimization objective is

$[answer2]$9

The reinforcement learning algorithm is GRPO. The appendix gives the objective as

$s_i$0

with token-level importance ratio

$s_i$1

For outcome-supervised training, every token in orchestration $s_i$2 receives the same group-normalized advantage: $s_i$3

These equations are not specific to MASBENCH alone, but they are the mechanism through which MAS-Orchestra is trained on MASBENCH’s controlled task distribution. The benchmark thereby serves as both an evaluation suite and a structured training environment for orchestration policies.

5. Empirical findings on MAS utility

The paper’s main empirical conclusion is that MAS gains are conditional rather than universal (Ke et al., 21 Jan 2026). With a weaker sub-agent backbone—Qwen-2.5-7B-Instruct used as the CoTAgent—MAS-Orchestra generally outperforms the SAS across most task structures except Depth. The Depth exception is one of the benchmark’s clearest findings: when the task is strongly sequential and interdependent, a single CoT chain may be more efficient because there is little to parallelize or modularize.

A second result is that stronger sub-agents shrink the marginal value of MAS. When the sub-agent is upgraded to GPT-OSS-120B in the low reasoning setting, MAS gains diminish substantially across Depth, Horizon, Breadth, and Parallel. The authors interpret this as evidence that once the sub-agent is strong enough to internalize decomposition or solve complex structures directly, coordination cost and error propagation offset the benefits of multi-agent decomposition.

Robustness is the regime in which MAS consistently help. In the adversarial setting, SAS accuracy collapses to near zero, while MAS retains substantially higher accuracy. The paper attributes this to structured decomposition, redundancy, cross-verification, and the ability to introduce moderation-like roles. In the qualitative example described in the paper, the MAS creates separate sub-agents for conflicting subtasks plus a final-answer agent acting like a moderator, whereas the SAS absorbs the incorrect adversarial note.

The benchmark also produces findings about orchestration quality. Holding the sub-agent fixed at GPT-120B (low), a Qwen-7B instruction-tuned orchestrator outperforms a GPT-20B reasoning orchestrator. The paper’s explanation is that reasoning models tend to solve the task themselves and under-delegate, sometimes collapsing toward one-agent designs, while instruction-tuned models are better aligned to decomposition, delegation, and workflow control.

Across axes, accuracy decreases as complexity increases, but out-of-distribution complexity values show the same trend as in-distribution ones. The authors interpret this as evidence that MASBENCH induces controlled complexity increases rather than arbitrary distribution shift.

6. Interpretation, deployment implications, and limitations

MASBENCH informs the paper’s deployment strategy for MAS-Orchestra by suggesting that MAS should be used selectively rather than uniformly (Ke et al., 21 Jan 2026). In downstream public benchmark evaluation, the authors choose low degree of MAS for AIME24, AIME25, and GPQA, which they view as largely sequential mathematical or reasoning tasks, and high degree of MAS for HotpotQA and BrowseComp+, which they view as more parallel or retrieval-heavy and closer to the edge of single-agent competence. This suggests a deployment rule based on structural cues and capability regime rather than on a blanket preference for multi-agent systems.

The benchmark also supports diagnostic interpretation. Strong gains on Parallel but not Depth indicate effective decomposition and concurrency with weak serial handoff efficiency. Gains on Breadth but not Horizon indicate strong aggregation of multiple predecessor results but weak long-range carrying of intermediate states. Strong performance on Robustness indicates verification, cross-checking, or moderator-style coordination. These interpretations are not formal benchmark outputs, but the paper explicitly presents MASBENCH as a controlled stress suite that enables such analysis.

Several limitations are acknowledged. The five axes do not exhaust all factors relevant to MAS utility; multi-stakeholder or negotiation settings are outside the benchmark. Robustness is instantiated only in practice on Depth $s_i$4 tasks, making that axis less comprehensive than the others. The benchmark is largely synthetic, which is a deliberate design choice for control but may omit messier real-world interaction patterns. The paper also notes that MAS-Orchestra currently does not observe intermediate execution states while planning, since holistic orchestration generates the complete orchestration in a single step.

Taken together, MASBENCH is best understood as a controlled benchmark for turning the question “Do multi-agent systems help?” into a structured empirical question indexed by dependency-graph structure, verification regime, orchestration policy, and sub-agent strength. Its main contribution is not a single benchmark score, but a framework for specifying when coordination is useful, when it is neutral, and when it is counterproductive.