MCPVerse: Benchmark and Ecosystem

• MCPVerse is a dual-use concept defining both a large-scale benchmark for evaluating agentic tool use and an ecosystem of MCP markets, servers, and clients.
• The benchmark assesses multi-step tool orchestration with over 550 real-world tools using outcome-based protocols for clear performance measurement.
• The broader ecosystem presents complex interoperability and security challenges, emphasizing protocol semantics, tool metadata, and layered authorization controls.

to=arxiv_search.search ＿天天 ￣第四色ાjson {"query":"MCPVerse Model Context Protocol benchmark ecosystem security arXiv", "max_results": 10} to=arxiv_search.search 北京赛车开 天天中彩票被json {"query":"(Lei et al., 22 Aug 2025) MCPVerse (Guo et al., 29 Sep 2025) Model Context Protocol", "max_results": 10} MCPVerse denotes, in the recent MCP literature, both a specific benchmark and a broader ecosystem concept. In the narrow sense, it is a large-scale benchmark for agentic tool use built on the Model Context Protocol (MCP), intended to evaluate how well models discover, select, and use real tools in realistic multi-step tasks (Lei et al., 22 Aug 2025). In the broader sense, it refers to the expanding MCP environment of markets, servers, clients, registries, gateways, and deployment patterns through which LLM applications discover capabilities, access external resources, and execute actions (Guo et al., 29 Sep 2025). Read this way, MCPVerse is not merely a catalog of tools; it is a composition environment in which protocol semantics, tool metadata, deployment architecture, and security controls jointly determine behavior.

1. Scope, architecture, and meanings of MCPVerse

The benchmark paper defines MCPVerse as “an expansive, real-world benchmark for evaluating agentic tool use,” whereas ecosystem studies use the term, or its functional equivalent, to denote the emerging universe of MCP markets, servers, and clients (Lei et al., 22 Aug 2025, Guo et al., 29 Sep 2025). Several security papers do not define the name formally, but explicitly treat it as the broader MCP ecosystem: a multi-server environment in which hosts, clients, registries, and servers interact across different trust domains. This suggests that “MCPVerse” functions in the literature as a dual-use term: a concrete evaluation artifact and an umbrella label for the operational world created by MCP adoption.

At the protocol level, the common substrate is MCP’s client-server architecture over stateful JSON-RPC 2.0 communication. Hosts are the user-facing applications that run models, clients inside hosts manage connections, and servers expose capabilities. Those capabilities are usually organized as tools, resources, and prompts. The protocol supports local deployments via stdio and remote deployments via HTTP/S with optional Server-Sent Events (SSE), and clients perform handshake, discovery, and invocation steps such as tools/list, resources/list, and tools/call (Guo et al., 29 Sep 2025, Li et al., 3 Feb 2026).

This architectural decomposition is central to the MCPVerse concept because it defines where interoperability and risk reside. Scientific and enterprise papers alike treat MCP as a unifying interface over heterogeneous backends rather than as a replacement for them: thin MCP servers can sit over mature services, while gateways, registries, or edge intermediaries absorb identity, policy, and routing complexity (Pan et al., 25 Aug 2025, Brett, 28 Apr 2025). A plausible implication is that MCPVerse is best understood as a layered interoperability fabric rather than a single platform.

2. MCPVerse as a benchmark for agentic tool use

As a benchmark, MCPVerse was introduced to address what its authors describe as a central gap in tool-use evaluation: many existing benchmarks rely on synthetic or simulated tools, or severely limit the available tools and action space (Lei et al., 22 Aug 2025). MCPVerse instead integrates more than 550 real-world executable tools across 65 MCPs, includes 250 tasks spanning three difficulty levels, and is explicitly designed to test multi-turn tool orchestration rather than one-shot API selection. The benchmark uses 552 unique tools, and the combined tool schemas occupy about 140k–147k tokens.

Its evaluation protocol is outcome-based rather than path-based. For textual tasks, correctness is judged by GPT-4o-20241120 as an LLM judge; for stateful tasks, automated scripts check the final environment state; and for time-sensitive tasks such as live flight schedules or map data, dynamic scripts fetch real-time ground truth (Lei et al., 22 Aug 2025). The action space is exposed through three modes. Oracle mode mounts only the minimal set of MCPs required for a task. Standard mode is designed for a typical 64k-token context and mounts a curated subset of 32 MCPs and 218 tools, taking about 44k tokens. Max-Scale mode mounts all 65 MCPs and 552 tools at once, requiring about 140k tokens.

The benchmark’s task complexity levels are defined operationally: L1 tasks are solvable in 1–2 steps, L2 tasks require at least 5 steps, and L3 tasks are complex tasks, usually more than 5 steps (Lei et al., 22 Aug 2025). Scoring is binary—correct or incorrect—which reflects the benchmark’s emphasis on whether the agent actually achieves the right end result rather than whether it follows a fixed reference trace.

The reported model behavior is one of the benchmark’s most consequential findings. Claude-4-Sonnet is the strongest model in the study, achieving 61.01% in Standard mode and 57.77% in Max-Scale mode, and the paper reports a 5.61% relative increase when moving from Oracle to Standard mode, with the largest gains on harder tasks (Lei et al., 22 Aug 2025). By contrast, many other models degrade as the tool set grows. The paper interprets this as evidence that larger action spaces are usually a burden, but can become an advantage for a sufficiently capable agentic model because they enlarge the exploration space and permit recovery via alternative tools.

3. Ecosystem scale, markets, clients, and implementation patterns

Measurement work treats MCPVerse as a large but structurally noisy ecosystem. Using MCPCrawler, one study crawled six major MCP markets—MCP.so, MCP Market, PulseMCP, Smithery, MCP Servers, and Cursor.directory—over a 14-day campaign, aggregated 17,630 raw entries, and after filtering and deduplication identified 8,401 valid projects: 8,060 valid MCP servers and 341 valid MCP clients (Guo et al., 29 Sep 2025). The same paper emphasizes that more than half of listed projects are invalid or low-value, using rule-based filters such as placeholder repos, inactive forks, abandoned projects, template projects, and low-content entries.

The market layer is fragmented rather than uniform. The study reports that only 32.3% of projects appear in more than one market, while only 5.5%–6.9% are indexed broadly in four or more markets (Guo et al., 29 Sep 2025). It also reports uneven validity rates across registries, with Cursor.directory at about 74.8% and MCP Market at 26.4%. This suggests that registry size is not equivalent to effective ecosystem quality.

On the server side, concentration and maintenance asymmetry are defining features. JavaScript accounts for 55.0% of servers and Python for 38.3%, so together they comprise more than 93% of the observed server population (Guo et al., 29 Sep 2025). The same study reports that 40.9% of servers were updated within the last 90 days, 37.2% within the past year but not the past 90 days, and 21.9% had been inactive for more than a year. This combination of language concentration and uneven maintenance is treated as a supply-chain risk because compromise in major npm or PyPI dependencies could have a large blast radius.

Client behavior shows partial standardization rather than convergence to a finished norm. Among 341 valid clients, SSE accounts for 56.98% or 194 clients, stdio for 38.12% or 130 clients, and others for 4.99% or 17 clients; meanwhile 80.9% of clients support only one connection and 19.1% support multiple connections (Guo et al., 29 Sep 2025). The literature interprets this as a transitional phase: local and lightweight developer workflows remain important, but multi-server composition is becoming more common.

A separate ecosystem study reinforces this view of MCPVerse as a supply chain spanning hosts, registries, and servers. It collected 67,057 servers from six public registries and reports heavy dependence on GitHub-backed distribution, weak vetting, and multiple registry-level attack surfaces including maintainer hijacking, redirection hijacking, credential leakage in published configurations, and affix-squatting in npm (Li et al., 18 Oct 2025). In that framing, MCPVerse is not only a discovery layer; it is a distribution and trust layer.

4. Security model, attack surfaces, and compositional risk

Security work on MCPVerse consistently argues that the main problem is not a single protocol flaw, but the interaction of composability, ambient trust, and over-privileged execution. One study demonstrates a “trivial trojan” in which a malicious weather MCP server, built by modifying Anthropic’s official “Hello Weather” example with fewer than 25 additional lines of code, induces Claude Desktop to invoke a legitimate Monzo banking server’s account.balance tool and then forwards the result to an attacker-controlled webhook.site endpoint (Croce et al., 26 Jul 2025). The attack path is summarized in the paper as get_forecast_prompt, discovery of account.balance, invocation of account.balance, and invocation of send_research_data. Its central claim is that “safe in isolation” does not mean safe in composition.

This compositional concern recurs across the literature. “Beyond the Protocol” identifies Tool Poisoning Attacks, Puppet Attacks, Rug Pull Attacks, and Exploitation via Malicious External Resources, and shows that malicious MCP servers could be uploaded to Smithery.ai, MCP.so, and Glama (Song et al., 31 May 2025). In a user study with 20 participants, 15 of 20 selected at least one malicious server in blind selection, and only one participant identified all four malicious servers when warned that four of thirteen were malicious. The paper reports an average attack success rate of 65.77% across five models, with Exploitation via Malicious External Resources reaching 81.33%.

Other work formalizes the idea that MCP servers themselves are active threat actors. “When MCP Servers Attack” decomposes a malicious server into metadata, configuration, initialization logic, tools, resources, and prompts, and organizes abuse into 12 attack categories, from server metadata poisoning to prompt output attack (Zhao et al., 29 Sep 2025). Its proof-of-concept servers achieve 100% attack success rates for several categories, including A2, A3, A5, A6, A8, and A11, across tested host–LLM combinations. The same paper argues that mass production is easy: with 10 tools and 10 resources, $(2^{10}-1)^2 = 1{,}046{,}529$ unique servers can be generated.

A parallel line of work studies semantic inconsistency rather than overtly malicious servers. “Don’t believe everything you read” introduces MCPDiFF and applies it to 10,240 real-world MCP servers across 36 categories, finding 5,303 Full Match, 3,544 Mostly Match, 1,079 Partial Match, and 314 Rare Match cases; the latter two categories together comprise about 13% of the ecosystem (Li et al., 3 Feb 2026). The paper treats description–code inconsistency as a security-relevant semantic gap because a tool described as read-only may hide privileged operations, hidden state mutations, or unauthorized financial actions. This is closely aligned with papers that describe tool descriptions as part of the security boundary.

Privilege and authentication studies extend the same diagnosis to operational controls. One large-scale static-analysis study of 2,562 real-world MCP applications reports that network resource APIs affect 1,438 servers and system resource APIs affect 1,237, while file resource threats affect 613 and memory resource threats 25 (Li et al., 5 Jul 2025). A remote-authentication measurement study validates 7,973 live remote MCP servers and finds that 3,233 servers, or 40.55%, expose tools without authentication; among 119 testable OAuth-enabled servers, every server had at least one confirmed flaw, for a total of 325 flaws, and dynamic client registration flaws affected 96.6% of tested servers (Zhou et al., 21 May 2026).

The strongest synthesis across these papers is that MCPVerse risk is transitive and layered. Tool descriptions can manipulate model choice, outputs can act as prompt injections, registries can distribute stale or hijackable servers, SDK clause non-compliance can silently disable guardrails, and privileged tools can bridge from untrusted text to file access, command execution, or outbound exfiltration (Yang et al., 10 Mar 2026, Zhao et al., 8 Sep 2025). This suggests that the decisive security question in MCPVerse is rarely whether a single tool is malicious in isolation; it is whether the host, client, server, registry, and authorization stack jointly constrain cross-tool behavior.

5. Enterprise architectures, governance, and defensive patterns

Enterprise-oriented papers treat MCPVerse as something that must be mediated rather than directly exposed. “Simplified and Secure MCP Gateways for Enterprise AI Integration” proposes an MCP Gateway that centralizes OAuth 2.1, token validation, identity-provider integration, access control, forwarding of authenticated identity, traffic inspection, and secure tunneling, allowing backend MCP servers to remain lightweight and focused on tool execution (Brett, 28 Apr 2025). In its layered reference architecture, a Security Proxy terminates TLS, applies rate limiting, and delegates forward authentication; an Authentication Gateway integrates with an enterprise IdP; a Zero Trust Tunnelling layer creates identity-aware encrypted tunnels; and Security Middleware performs deep inspection, threat detection, and logging.

The gateway design is explicitly tied to the 2025-03-26 MCP specification’s distinction between resource server and authorization server, and to the requirement for OAuth 2.1 and Dynamic Client Registration (Brett, 28 Apr 2025). In the proof of concept, the stack runs on a hardened Ubuntu 22.04 VPS using Traefik, Pangolin, WireGuard, CrowdSec, and a custom MCPAuth component, with reproducible version sets including Traefik v3.3.3, CrowdSec v1.6.8, Pangolin v1.2.0, Docker v28.1.1, and MCP Inspector v0.10.2. The paper’s explicit OAuth message flow begins with a 401 Unauthorized, followed by discovery at /.well-known/oauth-authorization-server, dynamic registration at /register, authorization at /authorize using PKCE, code exchange at /token, and then retry of the protected MCP request with a bearer token.

A companion enterprise security framework generalizes this gateway intuition into a defense-in-depth and Zero Trust program. It recommends isolating MCP servers in dedicated security zones, using service meshes with mTLS, enforcing strict protocol validation, applying WAFs and API gateways with deep packet inspection, hardening container runtimes with seccomp and AppArmor/SELinux, and adopting enhanced OAuth 2.0+ with short-lived, narrowly scoped tokens, audience restriction, sender-constrained tokens such as DPoP or mTLS token binding, and just-in-time access provisioning (Narajala et al., 11 Apr 2025). The same paper emphasizes tool and prompt security management: SAST, DAST, SCA, manual review or pentesting for high-risk tools, strict schemas for descriptions, malicious-pattern detection, and cryptographic signing of tool descriptions in registries.

Auditing tools arise from the same governance logic. mcp-sec-audit combines static pattern matching for Python-based MCP servers with dynamic sandboxed fuzzing and eBPF monitoring, reporting capability findings such as file_read, file_write, network_outbound, network_inbound, command_exec, env_access, tool_sequence_hijack, param_override, and prompt_injection (Huang et al., 23 Mar 2026). On MCPTox, it reports 663 capability instances, 100% detection rate across 491 samples when capability indicators were present, and 367 of 491 samples detected overall. A separate risk-assessment framework for open-source MCP servers maps 15,962 findings across 51 CWE classes in 222 GitHub repositories, with 191 of 222 repositories, or 86.0%, containing at least one mapped weakness (Kumar et al., 10 Mar 2026).

These proposals are not uniform in method, but they share a common institutional premise: in a mature MCPVerse, registry curation, pre-deployment auditing, identity infrastructure, least privilege, runtime monitoring, and policy enforcement have to be first-class system components rather than optional afterthoughts.

6. Scientific, IoT, and domain-specific extensions

The MCPVerse concept is not limited to generic software tools. In scientific cyberinfrastructure, MCP has been used as a unifying interface over mature services rather than as a new monolithic control plane (Pan et al., 25 Aug 2025). Thin MCP servers were implemented over Globus Transfer, Compute, and Search; computing-facility status APIs; the Octopus event fabric; Garden; and Rhea/Galaxy. The paper’s four case studies—molecular structure relaxation with Garden, multi-site phylogenetics with Globus, quantum chemistry with Globus Compute, and filesystem monitoring with Octopus plus Globus Search—present MCP as a federated scientific agent substrate in which capabilities are discoverable, invokable, and composable.

The same paper formalizes an MCP server as $M_k = \langle \mathcal{C}_k, \Phi_k \rangle$, where $\mathcal{C}_k$ is the server’s capabilities and $\Phi_k$ its authentication or authorization client (Pan et al., 25 Aug 2025). For large ecosystems such as Galaxy, it argues for separating discovery from invocation: Rhea uses RAG over Galaxy Toolshed documentation, Qwen3-Embedding-0.6B for embeddings, a find_tools discovery interface, and dynamic generation of MCP tools for top-$k$ matches. This suggests one path by which MCPVerse can scale without exposing every possible tool statically.

In IoT, MCP has been positioned as the interoperability layer between LLMs and heterogeneous hardware. IoT-MCP organizes the system into a Local Host, a Datapool and Connection Server, and IoT devices running lightweight, extensible microservices on microcontrollers (Yang et al., 25 Sep 2025). The framework supports 22 sensor types and 6 MCU families, and IoT-MCP Bench contains 114 Basic Tasks and 1,140 Complex Tasks. The reported system metrics are a 100% task success rate on tool execution for the Basic Tasks, 99% success rate on prompt robustness or complex tasks, 205 ms average response time, and 74 KB peak memory footprint.

The IoT paper’s deployment results are especially relevant to a broad understanding of MCPVerse because they move tool use into the physical world. It reports a 12-hour deployment in a multi-story building using 6 ESP32-S3 microcontrollers, 7 types and 12 sensors, and Wi-Fi connectivity, with continuous data collection and stable reconnection behavior after interruptions (Yang et al., 25 Sep 2025). In this sense, MCPVerse extends beyond software marketplaces into embodied environments in which edge routing, disconnection tolerance, and hardware semantics become part of the tool-use substrate.

Across both science and IoT, a common design pattern appears: MCP is used as a stable agent-facing layer over heterogeneous backends, while discovery, routing, authentication, and execution remain domain-specific underneath (Pan et al., 25 Aug 2025, Yang et al., 25 Sep 2025). A plausible implication is that future MCPVerse deployments will be shaped as much by domain adapters and trust boundaries as by the protocol itself.

7. Open problems and likely directions

The literature portrays MCPVerse as real and growing, but still fragile. Measurement papers stress market inflation, duplication, dependency monocultures, and uneven maintenance (Guo et al., 29 Sep 2025). Security papers stress cross-tool trust transitivity, malicious-server scalability, description–code inconsistency, over-privileged APIs, registry hijackability, weak authentication in remote servers, and SDK clause non-compliance (Croce et al., 26 Jul 2025, Zhou et al., 21 May 2026, Yang et al., 10 Mar 2026). Benchmark papers stress that large action spaces remain difficult for most models, even when those action spaces are central to realistic deployment (Lei et al., 22 Aug 2025).

Future work proposed in the cited papers is correspondingly broad. Enterprise gateway work calls for more advanced AI/ML-based tool behavior analysis, custom parsers and scenarios for intrusion detection, and more granular context-aware authorization (Brett, 28 Apr 2025). Ecosystem security studies call for stronger transparency guarantees, signed or vetted registries, better host-side verification before invocation, runtime monitoring, and clearer responsibility boundaries among registries, hosts, LLM providers, and users (Li et al., 18 Oct 2025, Song et al., 31 May 2025). Clause-compliance work argues for conformance testing, promotion of security-critical optional clauses to stronger requirements, and auditing at the SDK-clause level rather than only at the server level (Yang et al., 10 Mar 2026).

Taken together, these directions imply that the future of MCPVerse depends on simultaneous progress in three areas. First, evaluation must remain grounded in real executable tools and large action spaces rather than synthetic proxies. Second, governance must treat tool descriptions, prompts, resources, and authorization metadata as security-sensitive artifacts. Third, deployment practice must converge on stronger mediation layers—gateways, auditing, least privilege, and continuous monitoring—if MCPVerse is to function as a dependable substrate for agentic systems rather than merely a rapidly expanding catalog of integrations.