MCPCrawler: MCP Analysis & Code Exploration

• MCPCrawler is a family of systems that perform discovery, verification, measurement, and structured exploration of MCP artifacts across multiple platforms.
• It employs high-throughput crawling, multi-stage evidence verification, and normalization pipelines to build reproducible datasets and ensure quality.
• Its varied implementations—spanning ecosystem monitoring, repository analysis, and codebase knowledge graphs—support ecosystem governance, security auditing, and efficient code exploration.

MCPCrawler is a name applied in recent Model Context Protocol (MCP) literature to several technically distinct systems concerned with discovery, verification, measurement, and exploration of MCP artifacts. In one usage, it is the first end-to-end measurement framework designed to chart the MCP ecosystem at scale across public markets; in another, it is an end-to-end system for discovering, verifying, and harvesting real-world MCP implementations from GitHub; and, in Codebase-Memory, it denotes the implementation of a Tree-Sitter-based knowledge-graph system exposed as an MCP-compliant JSON-RPC server for code exploration (Guo et al., 29 Sep 2025, Toeppe et al., 11 Jul 2026, Vogel et al., 28 Mar 2026).

1. Terminological scope and disambiguation

The literature does not use MCPCrawler as a single, unique designation. Instead, the same name appears in at least three research contexts: ecosystem measurement, GitHub repository verification, and codebase knowledge-graph serving. This suggests that disambiguation by source paper is necessary when interpreting technical claims.

Usage of “MCPCrawler”	Source	Primary function
Measurement framework for the MCP ecosystem	(Guo et al., 29 Sep 2025)	Collects and normalizes data from six major markets
GitHub discovery and verification pipeline	(Toeppe et al., 11 Jul 2026)	Produces a reproducible dataset of MCP clients, servers, and gateways
Implementation of Codebase-Memory	(Vogel et al., 28 Mar 2026)	Exposes typed MCP tools for structural code exploration

The first two usages are explicitly data-collection systems. The third is a serving and analysis system: it implements an MCP-compliant JSON-RPC server, advertises 14 tools, and provides structured graph queries to an LLM agent. A common misconception is therefore to treat MCPCrawler as a single canonical software artifact. The cited work instead presents a family of systems sharing a name but operating at different layers of the MCP stack.

2. MCPCrawler as a measurement framework for MCP markets

In "A Measurement Study of Model Context Protocol" (Guo et al., 29 Sep 2025), MCPCrawler is presented as the first end-to-end measurement framework designed to chart the Model Context Protocol ecosystem at scale. Within that study, MCP is defined as a client–server JSON-RPC standard that lets LLMs mount external tools and data sources via MCP clients and servers. The framework’s primary goals are to discover and quantify the true scale of MCP marketplaces, to analyze the security and privacy posture of MCP servers, and to characterize MCP client interaction patterns.

The implementation is in Python and uses asyncio/aiohttp for high-throughput HTTP crawling, Playwright for headless browser rendering when markets employ dynamic JavaScript, and RabbitMQ as a message bus coordinating three modular subsystems. Those subsystems are the Market Adapter, the Server Resolver, and the Client Profiler. The Market Adapter is a plugin-based layer in which each market is encapsulated by a Python module providing fetch(), normalize(), and persist() interfaces. It uses aiohttp for REST/JSON-API markets such as Glama.ai and Smithery, Playwright plus headless Chromium for HTML-only sites such as MCP.so and Cursor.directory, rotating IP proxies to bypass rate limits, and a semi-automated CAPTCHA handler. The Server Resolver performs multi-feature entity matching across markets using GitHub URLs, TF–IDF cosine similarity on names and descriptions, author and license metadata, and temporal signals such as last commit. The Client Profiler aggregates GitHub popularity metrics, market-specific ratings, and usage counts, and parses each client’s transport handshake to classify supported protocols and connection multiplicity.

The pipeline is organized into four stages: discovery, fetching, validation, and normalization. Discovery enumerates pages or API listings and produces raw JSON or HTML blobs. Fetching downloads raw project metadata in parallel under per-market rate-limit profiles. Validation applies rule-based noise filters to exclude low-value entries such as those with only “Init commit” or a last update more than 12 months old. Normalization canonicalizes fields into a unified schema, merges cross-market duplicates, and computes composite quality scores for clients. The normalized schema includes common fields such as name, owner, URL, description, tags, license, last_update, and GitHub stars/[forks](https://www.emergentmind.com/topics/forks); server-specific fields such as declared dependencies, code size, LOC, and commit history; and client-specific fields such as transport protocols, max_connections, and composite_quality_score. After validation, entries are deduplicated via content hashing plus entity matching and stored in a normalized PostgreSQL schema.

The campaign targeted six major MCP registries—Smithery, MCP.so, Glama.ai, MCP Market, Cursor.directory, and PulseMCP—over a continuous 14-day crawl, with each market queried once per day. A centralized scheduler enqueues per-market tasks into RabbitMQ, and worker pools of up to 50 concurrent adapters issue asynchronous requests while respecting market-specific rate limits such as 120 req/min for MCP.so and 200 req/min for Glama.ai. Rotating IP pools and authenticated proxies minimize CAPTCHAs; on challenge, a Playwright worker resolves it using cached cookies.

3. Empirical findings from market-scale crawling

The principal quantitative result of the market study is that MCPCrawler aggregated 17,630 raw entries, of which 8,401 valid projects—8,060 servers and 341 clients—were analyzed (Guo et al., 29 Sep 2025). The paper defines the Valid Project Ratio as

$$R_{\mathit{valid}}=\frac{N_{\mathit{valid}}}{N_{\mathit{raw}}}.$$

Across the six markets, 49.1% of raw entries were deemed valid. The breakdown given in the study is 16,950 raw servers to 8,060 valid servers, and 363 raw clients to 341 valid clients. The abstract summarizes the implication more directly: more than half of listed projects are invalid or low-value.

The per-market validity distribution is uneven. MCP.so contributed 16,646 raw servers and 7,223 valid servers, corresponding to 43.4%. MCP Market contributed 14,280 raw entries and 3,765 valid entries, corresponding to 26.4%. Cross-market overlap is also limited: 32.3% of projects appear in at least two markets, but only 5.5% appear in at least four. This indicates that the ecosystem is not merely duplicated across registries; it is also fragmented across partially overlapping listings.

The server analysis emphasizes structural risks. Among Java servers, spring-boot, spring-core, and spring-web dominate. Python and TypeScript frequently use pydantic and zod for schema validation, whereas Go and Rust rely on generic serialization libraries such as serde and yaml.v3, lacking enforced validation. Maintenance is similarly skewed: 40.9% of servers were updated within 90 days, 37.2% were updated 90–365 days ago, and 21.9% were dormant for more than one year. The paper characterizes these conditions as dependency monocultures and uneven maintenance.

The client-side findings describe a transitional phase in protocol and connection patterns. Protocol support is led by SSE at 56.9% (194 clients), followed by stdio at 38.1% (130), with others at 4.9%. Connection multiplicity is dominated by single-server clients at 80.9% (276), while multi-server clients account for 19.1% (65). This supports the paper’s claim that clients exhibit a transitional phase in protocol and connection patterns rather than full convergence on one design.

The framework’s own performance evaluation reports average throughput of 147.6 entries/s, a peak of 181.3 entries/s on Smithery, and 122.1 entries/s on MCP.so; a 96.7% average per-query success rate, with failures never exceeding 5%; processing of 10.2 million requests and 17,630 raw entries in 14 days; and a noise-filtering accuracy of 93.5% true positive rate in a 500-entry manual audit. The study’s limitations are explicit: snapshot bias from a 14-day crawl, metadata scope that may miss runtime behaviors such as dynamic loading and undocumented APIs, classification boundaries that abstract over nuanced implementations, and extensibility issues as new markets emerge.

4. MCPCrawler as a GitHub discovery and verification pipeline

In "A Large-Scale Dataset of MCP Implementations on GitHub" (Toeppe et al., 11 Jul 2026), MCPCrawler is defined differently: it is an end-to-end system for discovering, verifying, and harvesting real-world MCP implementations from GitHub. The pipeline combines GitHub REST and GraphQL APIs with custom Python scripts and rule-based evidence tagging, and it outputs a high-precision, reproducible dataset of MCP clients, servers, and gateways.

The pipeline consists of five major stages. The first stage, Candidate Query, uses REST Search API queries such as “MCP in:name,readme,description” and “mcp server …” to retrieve up to 3,238 unique repositories, while GraphQL is used in bulk to fetch topics, languages, default branch, and archived or fork status. The second stage, Initial Filtering, removes forks, archived repositories, repositories with no commits in the last nine months, pure documentation or tutorial stubs, and forks without any local changes, leaving 3,058 repositories. The third stage, Multi-Stage Evidence Verify, loads each repository’s file tree and applies three layers of checks: manifest and dependency evidence, directory and entry-point evidence, and integration and environment evidence. Repositories failing all three layers are dropped; the paper specifies 660 non-operational repositories plus 11 scaffolds, leaving 2,387 for role classification. The fourth stage, Role Classification, uses a rule-based scoring system over client, server, and gateway signals. The fifth stage, Exclude Non-Op, removes templates, tutorials, demo repositories, and unrelated uses of “MCP,” including Minecraft Coder Pack.

The evidence model is central to the system. Manifest and dependency evidence searches for files such as package.json, pyproject.toml, go.mod, Cargo.toml, and composer.json, and for MCP-related dependencies such as @mcp/sdk, fastmcp, and mcp-client. Directory and entry-point evidence includes patterns such as /src/mcp/, /cmd/mcp/, and bin/mcp-server.js, or script entries such as mcp-server and mcp:run. Integration and environment evidence includes CI workflows invoking MCP tests or deployments, Dockerfiles or docker-compose references to MCP services, and configuration artifacts such as config.json for Claude Desktop. Each evidence type is tagged as strong when it appears in code and weak when it appears only in README or topic metadata; a repository is considered verified if at least one strong signal exists in any of the three verification layers.

The role classifier computes three scores, $S_{(c)}$, $S_{(s)}$, and $S_{(g)}$, by matching strong signals worth 2 points and weak signals worth 1 point. The paper gives the decision logic explicitly: a repository is classified as Client if $S_{(c)} \ge 3$ and $S_{(s)} \le 1$; as Server if $S_{(s)} \ge 3$ and $S_{(c)} \le 1$; and as Gateway if $S_{(g)} \ge 4$ and $S_{(c)}<2$ and $S_{(c)}$0. Otherwise it is marked “Low Confidence” and manually disambiguated or labeled Unclassified. A client override reassigns repositories with editor-extension signals such as vscode, cursor, and claude_desktop to Client even when gateway evidence is high.

Reproducibility is addressed through a frozen data-collection window from Jan 2024 to Oct 2025, publication of the JSONL export and schema on Zenodo DOI 10.5281/zenodo.17573071, documentation of exact search queries, filter rules, and evidence thresholds, and deterministic exclusion rules for tutorials and demos. Each JSONL record includes repository metadata, operational roles, subtype, evidence fields, and verification scores.

5. Verification metrics, dataset structure, and observed implementation patterns

The GitHub study reports a manual validation on a random sample of $S_{(c)}$1 from the 2,387 automatically verified repositories, reviewed independently by two authors (Toeppe et al., 11 Jul 2026). The observed rates are True Positives = 0.83, False Positives = 0.12, and False Negatives = 0.05. The paper uses the normal-approximation confidence interval

$S_{(c)}$2

and, substituting $S_{(c)}$3 and $S_{(c)}$4, reports a final pipeline precision of 83% with 95% confidence interval $S_{(c)}$5. Based on the review, 90 additional false matches were excluded, yielding 2,297 verified repositories.

The exported schema is newline-delimited JSON and includes fields such as repo_id, owner, name, url, created_at, pushed_at, stars, forks, languages, roles, subtype, evidence, and verification_score. The evidence object preserves the provenance of verification, including manifest files, entry points, directory patterns, CI configurations, and Dockerfiles. This schema is designed for reproducible downstream analysis rather than only one-off crawling.

The study’s language distribution shows that Python and TypeScript dominate MCP development. For servers, the reported counts are Python (785), TypeScript (597), and JavaScript (177). For clients, they are Python (530), TypeScript (492), and JavaScript (137). For gateways, they are TypeScript (25), Python (22), and Go (17). Hybrid architecture patterns are also described. The paper reports monorepos combining client and server code at approximately 8% of repositories, and microservice-style splits such as a Python server, a Node or TypeScript client, and a Go gateway.

A further qualitative finding is that manual review revealed repositories functioning primarily as educational samples, tutorials, or demonstration templates. The targeted exclusion of such repositories is not incidental; it is a methodological step required to separate operational implementations from non-operational examples. This mirrors, at repository scale, the market study’s distinction between raw entries and valid projects.

6. MCPCrawler as the implementation of Codebase-Memory

In "Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP" (Vogel et al., 28 Mar 2026), MCPCrawler refers to the implementation of Codebase-Memory rather than a registry crawler. It is an MCP-compliant JSON-RPC server that advertises 14 typed tools for indexing, query, analysis, and code, and serves a persistent, Tree-Sitter-based knowledge graph to an LLM agent. The system parses 66 languages through a multi-phase pipeline with parallel worker pools, call-graph traversal, impact analysis, and community detection.

The architecture comprises a file watcher using XXH3 plus polling, a Tree-Sitter parser pool, in-memory buffers (cbm_gbuf_t), a build stage, SQLite in WAL mode, and an MCP server. All six phases run inside a single SQLite transaction with deferred indexes. Phase 1 performs structure discovery and emits nodes such as Project, Package, Folder, and File. Phase 2 performs extraction, parsing ASTs and extracting Definition nodes—Function, Method, Class, Interface, Enum, and Type—together with attributes such as qualified_name, signature, file_path, start_line, export_flag, and cyclomatic_complexity. Phase 3 performs resolution and call-graph construction using a six-strategy cascade with confidence scores $S_{(c)}$6: import-map exact, import-map suffix, same-module prefix, unique simple-name, suffix-match with path distance, and fuzzy string match. Hybrid LSP passes for Go and C/C++ can bypass the cascade when calls are resolved directly. Phase 4 adds enrichment edges such as TESTS, HTTP_CALLS, ASYNC_CALLS, CONFIGURES, and FILE_CHANGES_WITH, and implements impact analysis through the detect_changes tool. Phase 5 flushes buffered nodes and edges to SQLite. Phase 6 runs Louvain modularity optimization on the CALLS+HTTP_CALLS+ASYNC_CALLS subgraph, using

$S_{(c)}$7

The knowledge-graph schema is explicitly typed. Node types include Project, Package, Folder, File, Module, Function or Method, Class, Interface, Enum, Type, Route, and Community. Edge types include CALLS, HTTP_CALLS, ASYNC_CALLS, IMPORTS, CONTAINS, DEFINES, DEFINES_METHOD, IMPLEMENTS, INHERITS, DECORATES, USES_TYPE, USAGE, THROWS, READS, WRITES, CONFIGURES, TESTS, FILE_CHANGES_WITH, and MEMBER_OF. In-memory buffers are keyed by label plus qualified_name for fast deduplication, and deferred SQLite indexes are created on (label, qualified_name) for nodes and on (type, src_id) and (type, dst_id) for edges.

The MCP communication pattern is conventional JSON-RPC. An LLM agent can call index_repository to queue parsing and graph construction, index_status to poll progress, and tools such as trace_call_path, query_graph, and detect_changes to query the resulting structure. The paper emphasizes that MCP ensures all calls and returns are strictly typed JSON, so the LLM never must ingest raw files and instead sees structured JSON results.

The evaluation covers 31 real-world repositories. Codebase-Memory reports 83% answer quality versus 92% for a file-exploration agent, at ten times fewer tokens and 2.1 times fewer tool calls. For graph-native queries such as hub detection and caller ranking, it matches or exceeds the explorer on 19 of 31 languages. Additional reported figures are tool calls per question of 2.3 for MCPCrawler versus 4.8 for the explorer, and query latency of less than 1 ms versus 10–30 s. The breakdown is not uniformly favorable: macro-heavy C yields MCPCrawler 0.58 versus Explorer 1.00, and the explorer wins in 16 of 31 cases when full source context or exhaustive grep is needed.

7. Research significance, limitations, and recurring themes

Across the cited work, MCPCrawler is consistently associated with large-scale, reproducible analysis of MCP artifacts, but the target artifact differs: public market listings, GitHub repositories, or repository-internal code structure. This suggests that the term has become attached to a methodological style—systematic collection, normalization, evidence verification, and structured querying—rather than to one standardized platform.

Several recurring findings are notable. First, raw MCP counts are not equivalent to operational systems. The market study reports that only 49.1% of raw entries were valid, while the GitHub study identifies and removes educational samples, tutorials, demos, scaffolds, and unrelated “MCP” repositories (Guo et al., 29 Sep 2025, Toeppe et al., 11 Jul 2026). Second, reproducibility is treated as a primary design constraint. The market study uses a normalized PostgreSQL schema and a documented 14-day campaign; the GitHub study freezes the collection window, publishes JSONL plus schema, and documents exact search queries and evidence thresholds; the Codebase-Memory work exposes typed JSON-RPC tools and persists its graph in SQLite WAL mode (Guo et al., 29 Sep 2025, Toeppe et al., 11 Jul 2026, Vogel et al., 28 Mar 2026). Third, each system identifies limits in static or metadata-based views: snapshot bias and metadata scope in the market study, precision bounds and non-operational samples in the GitHub study, and reduced quality on macro-heavy C or source-context-dependent tasks in Codebase-Memory.

The longer-term significance claimed most explicitly in the market study is ecosystem governance, security auditing, and standardization. Its conclusion states that large-scale, reproducible measurement of a heterogeneous MCP ecosystem is feasible, and that its public dataset and open-source code provide a foundation for ongoing ecosystem governance, security auditing, and standardization efforts (Guo et al., 29 Sep 2025). The GitHub dataset paper similarly frames its output as a foundational benchmark for studying real-world MCP ecosystems and for future research on integration, connectivity, and compatibility across the broader developer community (Toeppe et al., 11 Jul 2026). In the code-exploration setting, the significance is different but related: typed MCP tools and persistent knowledge graphs can reduce token consumption and tool-call frequency while preserving substantial answer quality for structural software-engineering queries (Vogel et al., 28 Mar 2026).