Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability

Abstract: The Model Context Protocol (MCP) has rapidly established itself as a standard interface for enabling LLM-based agents to interact with external tools and services. As MCP servers are increasingly entrusted with security-sensitive operations, understanding their real-world risks has become critical. In practice, due to the absence of large-scale runtime MCP servers, such understanding largely relies on security scanners applied to a small number of cases, yet the reliability of these assessments remains unclear. In this study, we revisit how MCP security is measured. We present MCPZoo, the largest collection of MCP servers for dynamic analysis to date. MCPZoo is constructed through a multi-agent framework for transforming in-the-wild static repositories into dynamic services. The framework emulates how human experts build, diagnose, and iteratively repair deployment and runtime defects by combining environment inference with feedback-driven refinement. To ensure practical interactivity at runtime, the servers are validated via real protocol interactions. As a result, MCPZoo contains 64,611 unique MCP servers (113,927 in total), with more than 37,288 supporting dynamic analysis. Leveraging MCPZoo, we conduct the first ecosystem-scale measurement of MCP servers and the scanners that analyze them. While existing scanners report that 96.89% of servers are risky, we find that these signals are unreliable. In particular, manual validation shows that less than 50% of sampled alerts are true positives, and scanner outputs exhibit clear inconsistency across scanners. Overall, MCPZoo enables large-scale, reproducible measurement of MCP server security and exposes limitations of current scanning practices. We further release a public query interface to support practical risk assessment of MCP servers.

• The paper introduces MCPZoo, an automated framework that transforms static MCP repositories into 64,611 runtime servers to enable comprehensive security analysis.
• The methodology reveals significant ecosystem issues, including over 90% code duplication and only 19.6% of projects achieving complete native deployment.
• The scanner evaluation shows low average precision (45.53%) and minimal cross-tool agreement, questioning the reliability of scanner-generated risk alerts.

Large-Scale Security Assessment of MCP Servers: Design, Measurement, and Limitations of Scanner-Based Risk Evaluation

MCPZoo Dataset Construction and Ecosystem Characterization

The Model Context Protocol (MCP) is an interface standard for enabling LLM-agent interaction with external tools, providing agents with the ability to invoke capabilities ranging from file access to network communication and system-level operations. Despite its centrality in autonomous agent architectures, prior security analyses were constrained by insufficient scale and lack of runtime deployments: risk measurement thus often depended on static code analysis or limited dynamic sampling, with minimal empirical validation.

The paper introduces MCPZoo: a fully automated, multi-agent framework that, for the first time, systematically transforms static MCP repositories into 64,611 unique, runtime-verifiable MCP servers (37,288 supporting dynamic analysis). The framework iteratively emulates expert deployment practices—including environment inference, failure diagnosis, and feedback-driven repair—thereby resolving practical deployment defects (missing artifacts, misconfigurations, external dependencies) that previously impeded ecosystem-wide reproducibility. The resulting corpus enables high-fidelity, protocol-level interaction for dynamic behavioral analysis.

Figure 1: Automated multi-agent workflow of MCPZoo for large-scale collection, build, deployment, and runtime validation.

A critical ecosystem finding is the prevalence of duplication and redundancy across MCP markets. Cross-market overlap frequently exceeds 90%; 28.3% of servers are duplicate code-derived entities resulting from forks, mirrors, or template-driven replication, severely inflating ecosystem-scale metrics. Only 46,327 represent unique codebases after de-duplication. Template reuse is especially pronounced: e.g., the remote-mcp-server-authless template generates 537 nearly identical deployments.

Figure 2: Intersection ratio matrix reveals dominant cross-market duplication and codebase overlap among MCP servers.

Deployment success is markedly limited by ecosystem engineering practices. Explicit Dockerfile presence is observed in only 21.4% of projects, and successful native deployment (with Dockerfile, README, Config present) occurs in merely 19.6%. The MCPZoo agent-driven pipeline achieves a global interaction rate of 57.7%, but failures persist, mainly due to external dependency requirements and source-level defects.

Figure 3: Distribution of GitHub stars across MCP servers illustrates the long-tailed popularity pattern and the prevalence of low-visibility projects.

Tool Exposure, Capability Distribution, and Domain-Specific Risk Surfaces

Analysis of runtime-exposed tool sets shows that “super-servers” may expose thousands of tools, but 85.53% provide fewer than 10, pointing to concentrated capability domains. High-risk capabilities constitute 37.66% of all tools, including command/script execution, file modification, and outbound data transfer. Notably, server function classifications exhibit distinct risk surface profiles: developer tool servers concentrate 83.15% of command execution tools, while web search, finance, and medical servers dominantly utilize remote info retrieval.

Figure 4: Distribution of tool capabilities within each functional server domain, highlighting domain-specific risk concentrations.

The mapping between server functionality and tool capability reveals non-uniform exposure: communication and e-commerce servers allocate the majority of tools to outbound data transfer (critical for data exfiltration scenarios), while media content servers disproportionately expose local file modification primitives.

Empirical Security Scanner Analysis: Reliability, Precision, and Cross-Consistency

The paper evaluates eight widely used MCP security scanners—spanning static, dynamic, and hybrid methods—across the 37,288 interactable MCPZoo servers. Raw results are alarmist: 96.89% of servers are flagged as risky by at least one scanner, irrespective of star-popularity stratification, and none are flagged by all scanners simultaneously.

A comprehensive precision and recall validation uncovers strong limitations. Manual review indicates an average precision rate of 45.53%, with extreme variation (low-performing scanner at 10.4%, “conservative” scanners >70%). CVE-based recall benchmarking shows that even the best-performing scanner identifies only 74.29% of known vulnerable servers, while several high-precision scanners fail to detect any CVE cases, evidencing the tradeoff between coverage and true positive rate.

Cross-scanner agreement is minimal: the average pairwise Jaccard similarity for flagged servers is 15.66% globally and <6% for specific vulnerability categories (prompt injection, command execution, data leakage). Scanners differ not only in flagged volume, but in reporting category distribution and sensitivity to capability and metadata patterns.

Figure 5: Low pairwise Jaccard similarity among scanner outputs evidences lack of cross-tool consensus on risk claims.

Figure 6: Category-level Jaccard similarity for prompt injection reveals severe inter-scanner divergence.

Figure 7: Category-level Jaccard similarity for command execution illustrates major inconsistency in scanner findings.

Figure 8: Category-level Jaccard similarity for data leakage further confirms unreliable cross-scanner overlap.

Root causes of false positives and negatives are linked to scanner design: metadata-based approaches conflate capability with vulnerability; LLM-driven dynamic scanning suffers from probe instability and failure misinterpretation; static code analysis overstates risk due to semantic path reconstruction without runtime exploitability verification; rule-based scanners rely on pattern matching that poorly generalizes to emergent vulnerabilities.

Practical and Theoretical Implications

The findings decisively reject the claim that scanner-reported risk prevalence accurately reflects ecosystem-level insecurity. Instead, systematic risks arise from structural ecosystem factors—duplication, template replication, poor deployment hygiene—that amplify attack surfaces and propagate vulnerabilities across markets. However, scanner-based measurement neither provides actionable ground truth nor reliably identifies the attack surface without concrete exploit validation.

This undermines the utility of scanner reports for ecosystem triage: treating scanner alerts as direct evidence of vulnerability leads to both overestimation and risk-masking. Scanner outputs are useful as triage signals, but actual security posture assessment requires systematic validation and runtime exploit testing. The release of a public query interface, mapping MCP servers to scanner and validation outcomes, supports more cautious, evidence-driven risk assessment.

Practically, the dataset and methodology are foundational for future MCP security research—supporting more reproducible measurements, validation-driven scanner development, and scalable dynamic benchmarking of defense strategies. Theoretical implications include new directions for protocol-hardening, agent capability management, and vulnerability propagation modeling in agent-tool ecosystems.

Conclusion

This work provides an authoritative ecosystem-scale assessment of MCP server security, supported by the MCPZoo corpus and systematic scanner evaluation. While the MCP ecosystem exhibits structural risk factors, scanner-reported risk claims are neither reliable nor sufficiently validated. This highlights the necessity for dynamic, empirical vulnerability analysis and trusted validation pipelines, shifting security assessment practice beyond scanner-oriented methodologies (2607.11086).