MASEval: Multi-Agent System Evaluation

• MASEval is a framework-agnostic evaluation library for LLM-based multi-agent systems that standardizes execution, tracing, and reporting across heterogeneous frameworks.
• It evaluates entire systems—encompassing model quality, agent orchestration, and communication protocols—shifting the focus from isolated models to integrated architectures.
• Its design leverages modular abstractions, trace-first evaluation, and clear separation of concerns to reduce implementation effort and enhance reproducibility.

Searching arXiv for MASEval and directly related evaluation-framework papers. MASEval is a framework-agnostic evaluation library for LLM-based multi-agent systems that treats the entire system as the unit of analysis rather than only the underlying model. It is motivated by the claim that, in agentic systems, benchmark outcomes depend not only on model quality but also on organizational topologies, communication protocols, memory architectures, role differentiation, orchestration logic, framework choice, tool-calling mechanisms, and error handling. MASEval is therefore positioned as an evaluation infrastructure layer—not a benchmark dataset and not an agent framework—that standardizes execution, tracing, and reporting across heterogeneous frameworks and benchmarks through thin adapters and a trace-first runtime (Emde et al., 9 Mar 2026).

1. Scope and evaluation target

MASEval addresses a problem created by the rapid growth of frameworks such as smolagents, LangGraph, AutoGen, CAMEL, and LlamaIndex. The paper argues that existing benchmarks are largely model-centric: they fix the agent scaffold and compare only models, which becomes inadequate once systems involve multiple agents, structured coordination, tools, users, environments, and framework-specific execution behavior. In this setting, public benchmark scores can conflate model quality with implementation decisions, including topology, orchestration logic, and error handling (Emde et al., 9 Mar 2026).

The central conceptual shift is from asking how well a model performs on a benchmark to asking how well a system performs on a benchmark. In the paper’s formulation, the evaluated object includes the agent(s), the framework, and the coordination logic. In execution terms, a system run may also include an Environment, one or more AgentAdapter-wrapped agents, an optional User simulator, one or more Evaluator objects, and ModelAdapters for simulators or LLM-based evaluators. This framing is the basis for the paper’s first design principle, System as unit of analysis, which states that the agent system is evaluated as a whole, enabling comparison of architectural choices and framework implementations, not just model capabilities (Emde et al., 9 Mar 2026).

A common misconception is that agentic benchmark scores primarily measure model quality. MASEval explicitly disputes that assumption. Its main empirical claim is that framework choice matters nearly as much as model choice when comparing models in a similar capability and cost tier. That claim is the paper’s main justification for moving evaluation from models to systems (Emde et al., 9 Mar 2026).

2. Design principles and core abstractions

The library is organized around five explicit design principles: System as unit of analysis, Bring your own (BYO), Infrastructure, not implementation, Separation of concerns, and Trace-first evaluation. The BYO principle means that no framework, model provider, or logging backend is privileged, and the core never imports framework-specific code; the paper states that this is enforced via CI. “Infrastructure, not implementation” means that MASEval provides orchestration, tracing, and lifecycle management while leaving tools, agent behavior, and evaluation metrics under user control. “Separation of concerns” distinguishes task definition, environment, agent logic, and evaluation so each can be varied independently. “Trace-first evaluation” means that all components log to a shared trace organized by component, including per-agent message histories, model usage, and tool calls (Emde et al., 9 Mar 2026).

Framework-agnosticism is achieved through a deliberately minimal universal interface. The paper emphasizes that agent systems can appear as Python library calls, command-line tools, containerized services, or remote endpoints, so MASEval does not assume a common internal structure. Instead, adding a new framework requires implementing only two methods: \_run_agent() and get_messages(). This is the key mechanism by which heterogeneous agent frameworks are normalized into a common evaluation runtime (Emde et al., 9 Mar 2026).

The standardization layer is expressed through seven core abstractions in maseval/core/.

Abstraction	Role	Key function
Task	Atomic evaluation unit	Query, environment data, criteria
Benchmark	Lifecycle orchestrator	Setup, execution, evaluation
Environment	Task state and tools	Exposes callable tools
AgentAdapter	Framework wrapper	Standard run and trace interface
User	User simulator	Multi-turn interaction
Evaluator	Metric computation	Filter traces, compute metrics
ModelAdapter	Provider interface	Tool calling and token tracking

These abstractions are complemented by the package structure: maseval/core/ for abstract base classes and runtime, maseval/interface/ for lazy-loaded adapters for frameworks, model providers, and logging, and maseval/benchmark/ for benchmark implementations. The overall architecture is intentionally thin: it standardizes runtime behavior and trace collection without forcing a specific agent implementation style (Emde et al., 9 Mar 2026).

3. Runtime architecture, tracing, and lifecycle management

The benchmark lifecycle has five phases: Setup, Execute, Collect, Evaluate, and Report. During setup, the runtime instantiates environment, tools, user simulator, agents, and evaluators, then registers all components for trace collection. Execution runs agents with customizable turn orchestration. Collection gathers traces, configurations, and system metadata such as git state and Python environment. Evaluation computes metrics from traces. Reporting stores a structured report containing task ID, status, traces, configuration, and results, then clears the registry for the next repetition. The lifecycle supports n_task_repeats, so each task can be executed multiple times (Emde et al., 9 Mar 2026).

Tracing is central to MASEval’s methodology. Each agent maintains an independent message history, preserving partial observability in multi-agent systems. Through TraceableMixin, agents log steps and message histories, models log input-output pairs, token usage, and latency, tools log invocations and outputs, and simulators log generation attempts and retries. Metric computation follows a two-stage pattern: filter_traces() extracts relevant evidence from raw traces, and __call__() computes metrics from those filtered traces. This makes the evaluation trace-driven rather than dependent on opaque post hoc judgments (Emde et al., 9 Mar 2026).

The runtime also includes a thread-safe ComponentRegistry for automatic registration of components returned by setup methods, plus optional manual registration. Extension points are exposed through benchmark hooks such as setup_environment(), setup_agents(), setup_user(), setup_evaluators(), run_agents(), and evaluate(). A callback system includes BenchmarkCallback, EnvironmentCallback, and AgentCallback, with built-in callbacks for progress bars and result logging, and support for experiment tracking, early stopping, or external integrations (Emde et al., 9 Mar 2026).

Error attribution is explicit. AgentError counts against the score because the agent violated its contract, whereas EnvironmentError and UserError are treated as infrastructure failures and excluded from scoring. The appendix further describes per-task deadlines via TaskContext, cooperative checkpoint-based timeout handling, timeout actions of skip, retry, or extend, and debug behavior controlled by fail_on_task_error=False for batch continuation or fail_on_task_error=True for fail-fast debugging. Parallelism is exposed through a num_workers parameter using a thread pool, with per-thread registries and locks around callback invocation and report aggregation (Emde et al., 9 Mar 2026).

4. Experimental protocol and controlled comparison

The paper’s main empirical study is a full factorial experiment across 3 frameworks, 3 models, and 3 benchmarks, yielding 27 configurations. For each benchmark, the study selects 2 domains and runs all tasks within each domain. The benchmarks are MACS with Travel and Mortgage, ConVerse with Travel Planning and Real Estate, and MultiAgentBench with Research and Bargaining. The reported metrics are partial goal success rate (pGSR) for MACS, robustness for ConVerse, defined as $1 - \text{ASR}$ on the security split, Task Completion Rate for MultiAgentBench Research, and scaled Task Score (TS) for MultiAgentBench Bargaining (Emde et al., 9 Mar 2026).

The compared models are GPT-5-mini, Gemini-3.0-Flash, and Claude Haiku 4.5. The compared frameworks are smolagents, LangGraph, and LlamaIndex, implemented through SmolAgentAdapter, LangGraphAgentAdapter, and LlamaIndexAgentAdapter. The paper characterizes smolagents as minimalist, code-based tool calling, LangGraph as state machine-based coordination with explicit state management, and LlamaIndex as a flexible single- and multi-agent framework. All three models are run with temperature = 1.0 and top-p = 1.0, because those settings are universally available across the providers compared in the study (Emde et al., 9 Mar 2026).

The experiment is designed to isolate framework effects while preserving benchmark realism. The authors state that they keep constant tool definitions, agent topologies, user simulations, environment dynamics, evaluation logic, and execution limits, including maximum user turns and step budgets. Where frameworks count steps differently, budgets are mapped accordingly. Non-agent models used by the benchmarks—such as judges, environment simulation, and attackers—are held constant with Gemini-3.0-Flash across all conditions. In ConVerse, the attacker agent is fixed and only the defending agent varies across frameworks (Emde et al., 9 Mar 2026).

At the same time, the study intentionally does not align framework internals. The paper explicitly states that it preserves built-in defaults such as system prompts, tool-mounting mechanisms, and error handling, and that it does not modify or align those internals across frameworks. This is a defining methodological choice: MASEval is meant to compare system implementations as practitioners would actually encounter them, not a hypothetical harmonized version with all framework-specific behavior stripped away (Emde et al., 9 Mar 2026).

5. Quantitative findings and trace-based error analysis

The headline finding is that framework choice has an impact comparable to model choice. Averaged over the six reported domains, the paper reports a mean range across models of 14.2 percentage points and a mean range across frameworks of 12.4 percentage points. The corresponding mean SD across models is 7.5 percentage points, while the mean SD across frameworks is 6.5 percentage points. Across the six domains, the range between frameworks exceeds that between models in two of the six domains. This is the primary quantitative basis for the claim that multi-agent evaluation must move from models to systems (Emde et al., 9 Mar 2026).

The domain-level scores illustrate that no framework dominates universally. On MACS Travel, Haiku 4.5 scores 90.4 with smolagents, 68.3 with LangGraph, and 59.5 with LlamaIndex. On ConVerse Travel Planning, Haiku 4.5 scores 86.2 with smolagents, 95.8 with LangGraph, and 94.7 with LlamaIndex. On MultiAgentBench Research, all systems are high-performing and variability is smaller: the table reports a cross-model mean range of 3.8 and a cross-framework mean range of 5.6. By contrast, ConVerse Travel Planning shows especially large framework variation, with a cross-framework mean range of 20.1, larger than the cross-model mean range of 16.8 (Emde et al., 9 Mar 2026).

The paper’s qualitative case study concerns GPT-5-mini with smolagents on MACS. The trace analysis attributes the poor score not to model quality alone but to a specific interaction between the model’s behavior, the framework’s tool-calling convention, the benchmark’s user-turn cap, and the error-message format. The paper reports that smolagents forces a tool call at every step, GPT-5-mini tends to over-engage the user with repeated clarification attempts, MACS caps clarifying questions at five, and once the cap is hit GPT-5-mini misinterprets the “max turns reached” error and retries the same tool call up to 23 times. These retries merely rephrase the question rather than changing strategy. The goals may still eventually be achieved, but often at $\ge 10 \times$ higher token consumption. The authors note that other models and frameworks do not exhibit this failure mode (Emde et al., 9 Mar 2026).

The infrastructure evaluation reports substantial implementation-effort reductions. For $\tau^2$-Bench, total implementation falls from 8,804 LoC to 3,793 LoC, a change of -5,011 or -56.9%, while interface code falls from 1,982 LoC to 343 LoC, a change of -82.7%. For ConVerse, total implementation falls from 2,098 LoC to 1,354 LoC, a change of -35.5%, while interface code falls from 778 LoC to 71 LoC, a change of -90.9%. The paper summarizes these reductions as 83–91% reduction in orchestration or interface code for benchmark consumers and 35–57% reduction in overall implementation effort for benchmark producers (Emde et al., 9 Mar 2026).

6. Position within evaluation research, limitations, and availability

MASEval occupies a distinct position within the broader “Eval” literature. MATEval is a Multi-Agent Text Evaluation framework in which evaluator agents, a feedback agent, and a summarizer agent discuss open-ended text and produce reports containing error type, location, explanation, and score (Li et al., 2024). MACEval is a Multi-Agent Continual Evaluation network for dynamic evaluation of large models through interviewer, interviewee, and supervisor roles, with in-process data generation and ACC-AUC-style longitudinal metrics (Chen et al., 12 Nov 2025). MAEGE is an automatic methodology for validating evaluation metrics for grammatical error correction using correction lattices and gold partial orders derived from human annotations (Choshen et al., 2018). Mage is a four-axis protocol for executable Unity game scene synthesis that separates compile success, runtime success, structural fidelity, and mechanism adherence (Liu et al., 8 May 2026). This juxtaposition suggests a useful distinction: some frameworks use multiple agents to perform evaluation, whereas MASEval evaluates multi-agent systems themselves (Emde et al., 9 Mar 2026).

The paper is explicit about limitations. Current integration breadth is 4 frameworks and 7 benchmarks, and broader coverage depends on community contributions. The architecture has been tested with up to five agents, but scaling to larger multi-agent systems remains unexplored. The minimal adapter interface is described as a lower bound; future frameworks with different memory management, asynchronous execution, or tool-calling behavior may require thicker adapters. The library also prioritizes flexibility over convenience, implying more manual setup than opinionated end-to-end evaluation tools. The reported experiment is limited to three frameworks, three models, and three benchmarks, with the compared models intentionally drawn from a similar cost and capability tier. Framework internals are intentionally left unaligned, so the results reflect idiomatic implementations rather than a fully harmonized comparison (Emde et al., 9 Mar 2026).

The library nevertheless includes mechanisms for reproducibility and cost-aware evaluation. Reports capture git state, system information, and package versions. The runtime supports n_task_repeats and records repeat_idx. For high-cost settings, the architecture includes AdaptiveTaskQueue and InformativeSubsetQueue, motivated by Item Response Theory and DISCO; the paper states that overall benchmark performance can sometimes be estimated within roughly 2 percentage points of full evaluation using only 1% of tasks. Availability is fully specified: MASEval is open source, distributed under the MIT license, installable with pip install maseval, documented at https://maseval.readthedocs.io, and hosted at https://github.com/parameterlab/MASEval (Emde et al., 9 Mar 2026).