MCP-Eval: Outcome Evaluation for Language Agents

• MCP-Eval is a standardized, outcome-oriented protocol that benchmarks language agents by focusing solely on the correctness of final answers.
• It employs an LLM-as-Judge with a fixed prompt to ensure reproducible evaluations that closely align with human annotation.
• By ignoring intermediate tool actions, MCP-Eval streamlines performance measurement into a direct pass rate reflecting real-world task success.

MCP-Eval is a standardized, outcome-oriented evaluation methodology for benchmarking language agents operating within the Model Context Protocol (MCP) ecosystem. Originating from the MCP-AgentBench project as described in "MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools," MCP-Eval defines a systematic protocol for measuring real-world task success of agentic systems interacting with MCP servers and tools, applying automated, LLM-based judgment at scale (Guo et al., 10 Sep 2025).

1. Formal Definition

MCP-Eval operates on the premise that agent evaluation should prioritize real-world outcome correctness over trajectory- or call-level details. For a benchmark instance indexed by $i$, let $Q_{\mathrm{final}}^{(i)}$ denote the user query, $A_{\mathrm{ref}}^{(i)}$ the reference answer vetted by human annotators, and $A_{\mathrm{model}}^{(i)}$ the answer generated by the evaluated agent $M_{\mathrm{eval}}$. Evaluation proceeds in two steps:

1. The LLM-as-Judge function, $L_{\mathrm{judge}}$, applies a prompt template $\mathcal{I}_{\mathrm{judge}}$ to the tuple $$(Q_{\mathrm{final}}^{(i)},A_{\mathrm{ref}}^{(i)},A_{\mathrm{model}}^{(i)})$$, outputting a binary decision $J^{(i)} \in \{\mathrm{Pass}, \mathrm{Fail}\}$:

$$J^{(i)} = L_{\mathrm{judge}}\bigl(Q_{\mathrm{final}}^{(i)},\;A_{\mathrm{ref}}^{(i)},\;A_{\mathrm{model}}^{(i)},\;\mathcal{I}_{\mathrm{judge}}\bigr)$$

2. The primary metric, Pass Rate, is computed as:

$Q_{\mathrm{final}}^{(i)}$0

where $Q_{\mathrm{final}}^{(i)}$1 is the benchmark (e.g., the 600-query suite of MCP-AgentBench), and $Q_{\mathrm{final}}^{(i)}$2 is the indicator function (Guo et al., 10 Sep 2025).

2. Evaluation Philosophy and Principles

MCP-Eval's guiding principle is outcome-orientation. Rather than evaluating the internal action sequences, e.g., the correctness or minimality of tool invocation trajectories, MCP-Eval considers only whether the agent's final answer achieves the user-defined success criteria as determined by the judge's comparison with a human-verified reference answer. Intermediate tool calls, their order, and the specifics of ReAct-style reasoning are decoupled from the scoring, with the system agnostic to strategy so long as the outcome is verifiably correct (Guo et al., 10 Sep 2025).

No additional normalization, weighting by subcategory, or partial credit is applied—all instances contribute equally to the aggregate score. The judge is insulated from low-level tool-call logs, using exclusively the (query, reference, candidate) triplet and established prompt template for assessment.

3. MCP-Eval in the MCP-AgentBench Testbed

The practical implementation of MCP-Eval is tightly integrated with the MCP-AgentBench infrastructure:

• The evaluation corpus comprises 600 systematically structured queries across six categories and varying interaction complexity, drawing on 33 operational MCP servers hosting 188 tools.
• Agents may interact with the underlying tool ecosystem using either ReAct or direct Tool-Calling via the unified mcprouter interface; all server monitoring and tool-call logging are retained for reproducibility but not exposed to the judge.
• The final user-facing answer, together with the relevant reference and the original query, is submitted to the LLM-as-Judge to determine pass/fail status.
• This separation provides protocol-level and agentic modularity: diverse action sequences are permitted as long as the agent state evolves towards the reference-conformant final answer (Guo et al., 10 Sep 2025).

4. Statistical Properties, Reliability, and Human Agreement

Reliability of MCP-Eval is established through comparison to human annotation. On a random subset of 60 instances:

• Percentage agreement between MCP-Eval's LLM-based judge and human majority is $Q_{\mathrm{final}}^{(i)}$3.
• Cohen’s Kappa for judge vs. human majority is $Q_{\mathrm{final}}^{(i)}$4.
• Fleiss’ Kappa among human annotators is $Q_{\mathrm{final}}^{(i)}$5.
• Full three-way agreement among human annotators is $Q_{\mathrm{final}}^{(i)}$6.

These consistency figures demonstrate that MCP-Eval closely matches expert assessment and has practical inter-rater reliability suitable for outcome evaluation at scale (Guo et al., 10 Sep 2025).

5. Distinguishing Features Relative to Related Methodologies

MCP-Eval represents a clear methodological advance over prior MCP/agent benchmarks by imposing an outcome-centric standard:

• Trajectory-oriented metrics (e.g., tool-call matching, order, minimality) are purposefully excluded; correctness is determined solely by the realization of reference-level outcomes.
• The judgment mechanism is transparent, reproducible, and based on a fixed LLM + prompt configuration, with code and logs released to facilitate independent verification and downstream review.
• No category weighting, difficulty normalization, or partial credit is utilized; Pass Rate is a direct reflection of binary task success across the full testbed.
• This approach contrasts with frameworks such as MCPEval (Liu et al., 17 Jul 2025), MCPMark (Wu et al., 28 Sep 2025), and MCP-Atlas (Bandi et al., 31 Jan 2026), which rely on richer path-level, claims-based, or partial-credit metrics.

6. Implications, Limitations, and Role in the MCP Ecosystem

MCP-Eval sets a replicable baseline for outcome-focused benchmarking in the context of MCP-mediated agent-tool interaction. Its uniformity and simplicity enable large-scale comparative studies, reveal operational strengths and deficiencies of diverse agentic systems, and inform broader efforts to standardize agent evaluation. However, by abstracting away intermediate behaviors, it may omit diagnostic granularity necessary for agent debugging, optimization of planning strategies, or the understanding of failure modalities beyond outright outcome correctness (Guo et al., 10 Sep 2025).

No explicit confidence measures, statistical hypothesis tests, or error-bar computations are reported; the focus is on pass/fail judgment as the atomic unit of evaluation. A plausible implication is that for models or tasks where path-level explanation is crucial, MCP-Eval may need to be complemented by path-sensitive diagnostics.

7. Broader Adoption and Extensions

MCP-Eval's outcome-oriented design, reproducible protocol specification, and integration with MCP-AgentBench have positioned it as a reference standard in the MCP evaluation landscape. Reproducible implementations and data releases enable both adoption and critical scrutiny from the research community. Extensions may involve expansion to process-level metrics (as in MedMCP-Calc (Zhu et al., 30 Jan 2026)), richer diagnostic scoring, or adaptation to new agent models and tool ecosystems as MCP infrastructure evolves. The approach also harmonizes with recent directions in the LLM evaluation literature emphasizing LLM-as-Judge architectures and large-scale, outcome-level agent assessment.