---
title: 'MASEval: Multi-Agent System Evaluation'
url: https://www.emergentmind.com/topics/maseval
type: topic
---

# MASEval: Multi-Agent System Evaluation

Searching arXiv for MASEval and directly related evaluation-framework papers.
MASEval is a framework-agnostic evaluation library for LLM-based multi-agent systems that treats the **entire system** as the unit of analysis rather than only the underlying model. It is motivated by the claim that, in agentic systems, benchmark outcomes depend not only on model quality but also on organizational topologies, communication protocols, memory architectures, role differentiation, orchestration logic, framework choice, tool-calling mechanisms, and error handling. MASEval is therefore positioned as an **evaluation infrastructure layer**—not a benchmark dataset and not an agent framework—that standardizes execution, tracing, and reporting across heterogeneous frameworks and benchmarks through thin adapters and a trace-first runtime [2603.08835].

## 1. Scope and evaluation target

MASEval addresses a problem created by the rapid growth of frameworks such as **smolagents, LangGraph, AutoGen, CAMEL, and LlamaIndex**. The paper argues that existing benchmarks are largely **model-centric**: they fix the agent scaffold and compare only models, which becomes inadequate once systems involve multiple agents, structured coordination, tools, users, environments, and framework-specific execution behavior. In this setting, public benchmark scores can conflate model quality with implementation decisions, including **topology**, **orchestration logic**, and **error handling** [2603.08835].

The central conceptual shift is from asking how well a model performs on a benchmark to asking how well a **system** performs on a benchmark. In the paper’s formulation, the evaluated object includes the **agent(s)**, the **framework**, and the **coordination logic**. In execution terms, a system run may also include an **Environment**, one or more **AgentAdapter**-wrapped agents, an optional **User** simulator, one or more **Evaluator** objects, and **ModelAdapters** for simulators or LLM-based evaluators. This framing is the basis for the paper’s first design principle, **System as unit of analysis**, which states that the agent system is evaluated as a whole, enabling comparison of architectural choices and framework implementations, not just model capabilities [2603.08835].

A common misconception is that agentic benchmark scores primarily measure model quality. MASEval explicitly disputes that assumption. Its main empirical claim is that **framework choice matters nearly as much as model choice** when comparing models in a similar capability and cost tier. That claim is the paper’s main justification for moving evaluation from models to systems [2603.08835].

## 2. Design principles and core abstractions

The library is organized around five explicit design principles: **System as unit of analysis**, **Bring your own (BYO)**, **Infrastructure, not implementation**, **Separation of concerns**, and **Trace-first evaluation**. The BYO principle means that no framework, model provider, or logging backend is privileged, and the core never imports framework-specific code; the paper states that this is enforced via CI. “Infrastructure, not implementation” means that MASEval provides orchestration, tracing, and lifecycle management while leaving tools, agent behavior, and evaluation metrics under user control. “Separation of concerns” distinguishes task definition, environment, agent logic, and evaluation so each can be varied independently. “Trace-first evaluation” means that all components log to a shared trace organized by component, including per-agent message histories, model usage, and tool calls [2603.08835].

Framework-agnosticism is achieved through a deliberately minimal universal interface. The paper emphasizes that agent systems can appear as Python library calls, command-line tools, containerized services, or remote endpoints, so MASEval does not assume a common internal structure. Instead, adding a new framework requires implementing only two methods: `\_run_agent()` and `get_messages()`. This is the key mechanism by which heterogeneous agent frameworks are normalized into a common evaluation runtime [2603.08835].

The standardization layer is expressed through seven core abstractions in `maseval/core/`.

| Abstraction | Role | Key function |
|---|---|---|
| `Task` | Atomic evaluation unit | Query, environment data, criteria |
| `Benchmark` | Lifecycle orchestrator | Setup, execution, evaluation |
| `Environment` | Task state and tools | Exposes callable tools |
| `AgentAdapter` | Framework wrapper | Standard run and trace interface |
| `User` | User simulator | Multi-turn interaction |
| `Evaluator` | Metric computation | Filter traces, compute metrics |
| `ModelAdapter` | Provider interface | Tool calling and token tracking |

These abstractions are complemented by the package structure: **`maseval/core/`** for abstract base classes and runtime, **`maseval/interface/`** for lazy-loaded adapters for frameworks, model providers, and logging, and **`maseval/benchmark/`** for benchmark implementations. The overall architecture is intentionally thin: it standardizes runtime behavior and trace collection without forcing a specific agent implementation style [2603.08835].

## 3. Runtime architecture, tracing, and lifecycle management

The benchmark lifecycle has five phases: **Setup**, **Execute**, **Collect**, **Evaluate**, and **Report**. During setup, the runtime instantiates environment, tools, user simulator, agents, and evaluators, then registers all components for trace collection. Execution runs agents with customizable turn orchestration. Collection gathers traces, configurations, and system metadata such as git state and Python environment. Evaluation computes metrics from traces. Reporting stores a structured report containing task ID, status, traces, configuration, and results, then clears the registry for the next repetition. The lifecycle supports **`n_task_repeats`**, so each task can be executed multiple times [2603.08835].

Tracing is central to MASEval’s methodology. Each agent maintains an **independent message history**, preserving partial observability in multi-agent systems. Through `TraceableMixin`, agents log steps and message histories, models log input-output pairs, token usage, and latency, tools log invocations and outputs, and simulators log generation attempts and retries. Metric computation follows a two-stage pattern: **`filter_traces()`** extracts relevant evidence from raw traces, and **`__call__()`** computes metrics from those filtered traces. This makes the evaluation trace-driven rather than dependent on opaque post hoc judgments [2603.08835].

The runtime also includes a **thread-safe `ComponentRegistry`** for automatic registration of components returned by setup methods, plus optional manual registration. Extension points are exposed through benchmark hooks such as `setup_environment()`, `setup_agents()`, `setup_user()`, `setup_evaluators()`, `run_agents()`, and `evaluate()`. A callback system includes **`BenchmarkCallback`**, **`EnvironmentCallback`**, and **`AgentCallback`**, with built-in callbacks for progress bars and result logging, and support for experiment tracking, early stopping, or external integrations [2603.08835].

Error attribution is explicit. **`AgentError`** counts against the score because the agent violated its contract, whereas **`EnvironmentError`** and **`UserError`** are treated as infrastructure failures and excluded from scoring. The appendix further describes per-task deadlines via `TaskContext`, cooperative checkpoint-based timeout handling, timeout actions of **skip, retry, or extend**, and debug behavior controlled by `fail_on_task_error=False` for batch continuation or `fail_on_task_error=True` for fail-fast debugging. Parallelism is exposed through a **`num_workers`** parameter using a thread pool, with per-thread registries and locks around callback invocation and report aggregation [2603.08835].

## 4. Experimental protocol and controlled comparison

The paper’s main empirical study is a **full factorial experiment across 3 frameworks, 3 models, and 3 benchmarks**, yielding **27 configurations**. For each benchmark, the study selects **2 domains** and runs all tasks within each domain. The benchmarks are **MACS** with **Travel** and **Mortgage**, **ConVerse** with **Travel Planning** and **Real Estate**, and **MultiAgentBench** with **Research** and **Bargaining**. The reported metrics are **partial goal success rate (pGSR)** for MACS, **robustness** for ConVerse, defined as $1 - \text{ASR}$ on the security split, **Task Completion Rate** for MultiAgentBench Research, and **scaled Task Score (TS)** for MultiAgentBench Bargaining [2603.08835].

The compared models are **GPT-5-mini**, **Gemini-3.0-Flash**, and **Claude Haiku 4.5**. The compared frameworks are **smolagents**, **LangGraph**, and **LlamaIndex**, implemented through **`SmolAgentAdapter`**, **`LangGraphAgentAdapter`**, and **`LlamaIndexAgentAdapter`**. The paper characterizes smolagents as **minimalist, code-based tool calling**, LangGraph as **state machine-based coordination with explicit state management**, and LlamaIndex as a **flexible single- and multi-agent framework**. All three models are run with **temperature = 1.0** and **top-p = 1.0**, because those settings are universally available across the providers compared in the study [2603.08835].

The experiment is designed to isolate framework effects while preserving benchmark realism. The authors state that they keep constant **tool definitions**, **agent topologies**, **user simulations**, **environment dynamics**, **evaluation logic**, and **execution limits**, including maximum user turns and step budgets. Where frameworks count steps differently, budgets are mapped accordingly. Non-agent models used by the benchmarks—such as **judges**, **environment simulation**, and **attackers**—are held constant with **Gemini-3.0-Flash across all conditions**. In ConVerse, the attacker agent is fixed and only the defending agent varies across frameworks [2603.08835].

At the same time, the study intentionally does **not** align framework internals. The paper explicitly states that it preserves built-in defaults such as **system prompts**, **tool-mounting mechanisms**, and **error handling**, and that it does not modify or align those internals across frameworks. This is a defining methodological choice: MASEval is meant to compare system implementations as practitioners would actually encounter them, not a hypothetical harmonized version with all framework-specific behavior stripped away [2603.08835].

## 5. Quantitative findings and trace-based error analysis

The headline finding is that **framework choice has an impact comparable to model choice**. Averaged over the six reported domains, the paper reports a **mean range across models** of **14.2 percentage points** and a **mean range across frameworks** of **12.4 percentage points**. The corresponding **mean SD across models** is **7.5 percentage points**, while the **mean SD across frameworks** is **6.5 percentage points**. Across the six domains, the range between frameworks exceeds that between models in **two of the six domains**. This is the primary quantitative basis for the claim that multi-agent evaluation must move from models to systems [2603.08835].

The domain-level scores illustrate that no framework dominates universally. On **MACS Travel**, **Haiku 4.5** scores **90.4** with **smolagents**, **68.3** with **LangGraph**, and **59.5** with **LlamaIndex**. On **ConVerse Travel Planning**, **Haiku 4.5** scores **86.2** with **smolagents**, **95.8** with **LangGraph**, and **94.7** with **LlamaIndex**. On **MultiAgentBench Research**, all systems are high-performing and variability is smaller: the table reports a **cross-model mean range** of **3.8** and a **cross-framework mean range** of **5.6**. By contrast, **ConVerse Travel Planning** shows especially large framework variation, with a **cross-framework mean range** of **20.1**, larger than the **cross-model mean range** of **16.8** [2603.08835].

The paper’s qualitative case study concerns **GPT-5-mini with smolagents on MACS**. The trace analysis attributes the poor score not to model quality alone but to a specific interaction between the model’s behavior, the framework’s tool-calling convention, the benchmark’s user-turn cap, and the error-message format. The paper reports that smolagents **forces a tool call at every step**, GPT-5-mini tends to **over-engage the user with repeated clarification attempts**, MACS caps clarifying questions at **five**, and once the cap is hit GPT-5-mini misinterprets the **“max turns reached”** error and retries the same tool call up to **23 times**. These retries merely rephrase the question rather than changing strategy. The goals may still eventually be achieved, but often at **$\ge 10 \times$** higher token consumption. The authors note that other models and frameworks do not exhibit this failure mode [2603.08835].

The infrastructure evaluation reports substantial implementation-effort reductions. For **\(\tau^2\)-Bench**, total implementation falls from **8,804** LoC to **3,793** LoC, a change of **-5,011** or **-56.9%**, while interface code falls from **1,982** LoC to **343** LoC, a change of **-82.7%**. For **ConVerse**, total implementation falls from **2,098** LoC to **1,354** LoC, a change of **-35.5%**, while interface code falls from **778** LoC to **71** LoC, a change of **-90.9%**. The paper summarizes these reductions as **83–91% reduction** in orchestration or interface code for benchmark consumers and **35–57% reduction** in overall implementation effort for benchmark producers [2603.08835].

## 6. Position within evaluation research, limitations, and availability

MASEval occupies a distinct position within the broader “Eval” literature. **MATEval** is a **Multi-Agent Text Evaluation framework** in which evaluator agents, a feedback agent, and a summarizer agent discuss open-ended text and produce reports containing error type, location, explanation, and score [2403.19305]. **MACEval** is a **Multi-Agent Continual Evaluation network** for dynamic evaluation of large models through interviewer, interviewee, and supervisor roles, with in-process data generation and **ACC-AUC**-style longitudinal metrics [2511.09139]. **MAEGE** is an automatic methodology for validating evaluation metrics for grammatical error correction using correction lattices and gold partial orders derived from human annotations [1804.11225]. **Mage** is a four-axis protocol for executable Unity game scene synthesis that separates compile success, runtime success, structural fidelity, and mechanism adherence [2605.07342]. This juxtaposition suggests a useful distinction: some frameworks use multiple agents **to perform evaluation**, whereas MASEval evaluates **multi-agent systems themselves** [2603.08835].

The paper is explicit about limitations. Current integration breadth is **4 frameworks** and **7 benchmarks**, and broader coverage depends on community contributions. The architecture has been **tested with up to five agents**, but scaling to larger multi-agent systems remains unexplored. The minimal adapter interface is described as a **lower bound**; future frameworks with different memory management, asynchronous execution, or tool-calling behavior may require thicker adapters. The library also prioritizes **flexibility over convenience**, implying more manual setup than opinionated end-to-end evaluation tools. The reported experiment is limited to **three frameworks**, **three models**, and **three benchmarks**, with the compared models intentionally drawn from a similar cost and capability tier. Framework internals are intentionally left unaligned, so the results reflect idiomatic implementations rather than a fully harmonized comparison [2603.08835].

The library nevertheless includes mechanisms for reproducibility and cost-aware evaluation. Reports capture **git state**, **system information**, and **package versions**. The runtime supports **`n_task_repeats`** and records **`repeat_idx`**. For high-cost settings, the architecture includes **`AdaptiveTaskQueue`** and **`InformativeSubsetQueue`**, motivated by Item Response Theory and **DISCO**; the paper states that overall benchmark performance can sometimes be estimated within roughly **2 percentage points** of full evaluation using only **1% of tasks**. Availability is fully specified: MASEval is **open source**, distributed under the **MIT license**, installable with **`pip install maseval`**, documented at **`https://maseval.readthedocs.io`**, and hosted at **`https://github.com/parameterlab/MASEval`** [2603.08835].

Source: https://www.emergentmind.com/topics/maseval