Coverage-Aware Web Crawling for Domain-Specific Supplier Discovery via a Web--Knowledge--Web Pipeline

Abstract: Identifying the full landscape of small and medium-sized enterprises (SMEs) in specialized industry sectors is critical for supply-chain resilience, yet existing business databases suffer from substantial coverage gaps -- particularly for sub-tier suppliers and firms in emerging niche markets. We propose a \textbf{Web--Knowledge--Web (W$\to$K$\to$W)} pipeline that iteratively (1)~crawls domain-specific web sources to discover candidate supplier entities, (2)~extracts and consolidates structured knowledge into a heterogeneous knowledge graph, and (3)~uses the knowledge graph's topology and coverage signals to guide subsequent crawling toward under-represented regions of the supplier space. To quantify discovery completeness, we introduce a \textbf{coverage estimation framework} inspired by ecological species-richness estimators (Chao1, ACE) adapted for web-entity populations. Experiments on the semiconductor equipment manufacturing sector (NAICS 333242) demonstrate that the W$\to$K$\to$W pipeline achieves the highest precision (0.138) and F1 (0.118) among all methods using the same 213-page crawl budget, building a knowledge graph of 765 entities and 586 relations while reaching peak recall by iteration~3 with only 112 pages.

• The paper introduces a Web–Knowledge–Web pipeline that extracts entities into a knowledge graph, detects discovery gaps, and prioritizes new crawl seeds under a fixed HTTP budget.
• The pipeline achieved the highest precision (13.8%) and F1 score (11.8%) in a semiconductor-equipment supplier test, while reaching peak recall after 112 of 213 pages.
• Coverage estimation using Chao1 and ACE provided relative convergence signals but overestimated completeness, with wide confidence intervals and only 10.3% verified supplier recall.

Motivation and problem statement

The paper addresses the problem of discovering the full population of small and medium-sized enterprise (SME) suppliers in a specialized industrial domain, motivated by the poor coverage of sub-tier and niche suppliers in commercial procurement databases. The authors formalize discovery as maximizing the number of unique supplier entities found under an HTTP request budget, while additionally estimating what fraction of the true population has been found—a coverage-awareness goal that distinguishes this work from prior focused crawling, which treats the task as document retrieval without any notion of completeness.

The proposed Web–Knowledge–Web ($\text{W} \to \text{K} \to \text{W}$) pipeline iterates three phases: (1) LLM-based extraction of typed entities and relations from crawled pages into a heterogeneous knowledge graph (KG); (2) gap analysis on the KG—structural hole detection, query expansion, and link prediction—to generate targeted crawl seeds; and (3) priority-queue re-crawling scored by relevance, novelty, and gap association. The intermediate KG thus serves simultaneously as memory of what has been discovered and as a guide for what to crawl next.

Methodology

Extraction (Web → Knowledge). Each page is processed by GPT-4o-mini in a single zero-shot call constrained by a JSON schema to four entity types (COMPANY, PRODUCT, SECTOR, LOCATION) and five relation types (supplies_to, partners_with, produces, located_in, belongs_to_sector). Post-processing enforces schema-level type constraints and filters relations below confidence $\theta_r = 0.3$. Entity resolution uses name-prefix blocking with Jaro–Winkler similarity plus attribute overlap, merging pairs above $\theta_m = 0.85$. Notably, the total LLM API cost for 213 pages was approximately $0.06 USD, making the approach extremely inexpensive at this scale.

Gap-guided seed generation (Knowledge → Web). Three complementary signals drive re-crawling: degree-anomaly detection on sector nodes relative to industry priors, identification of missing bridge nodes in supplies_to paths, and geographic gap analysis. These are supplemented by a DistMult KG embedding ($d=100$, 50 epochs per iteration) that scores likely missing supplies_to edges and converts high-confidence predictions into additional search queries.

Coverage estimation. The paper adapts ecological species-richness estimators—Chao1 and ACE—to web-entity populations, treating each crawl iteration as a capture occasion and each entity as a captured individual. Chao1's estimate $$\widehat{|\mathcal{E}^*|} = |\hat{\mathcal{E}}| + f_1^2/(2f_2)$$, where $f_1$ and $f_2$ are singleton and doubleton counts across iterations or sources, yields a coverage ratio $\hat{C}$ used as a principled stopping criterion ($\hat{C} > \tau$ sustained over two iterations), with bootstrap confidence intervals computed over the source–entity incidence matrix. To the authors' knowledge, this is the first application of capture–recapture estimation to domain-specific entity discovery via crawling.

Experimental results

Evaluation targets NAICS 333242 (semiconductor equipment manufacturing), starting from 48 hand-curated seeds against an approximate ground truth of ${\sim}200$ companies assembled from SEMI member lists, VLSI Research rankings, trade-show directories, and manual curation. All methods share the same 213-page budget and extraction pipeline, differing only in crawl strategy:

Method	Discovered	TP	Precision	Recall	F1
BFS Crawler	160	20	0.125	0.103	0.113
Focused Crawler	236	18	0.076	0.092	0.084
W→K (no feedback)	160	20	0.125	0.103	0.113
W→K→W	145	20	0.138	0.103	0.118

Three findings deserve emphasis. First, BFS and W→K produce identical results, which the authors interpret as evidence that extraction alone confers no advantage—the K→W feedback loop is the sole differentiator. Second, W→K→W achieves the highest precision by returning fewer false positives (145 vs.\ 160–236 discovered entities), though recall is identical across three of four methods (20 TPs). Third, the conclusion claims the pipeline "discovers 185% more unique supplier companies than single-pass baselines (157 vs.\ 55)"—this figure compares cumulative company counts across all five iterations against iteration-1 output only, and is therefore not directly comparable to the fixed-budget table results, where recall differences vanish. Readers should treat these two headline numbers as measuring different things.

Peak recall was reached by iteration 3 using only 112 pages, supporting an efficiency claim relative to baselines exhausting the full 213-page budget. The final KG contains 765 entities and 586 relations. A case study attributes three discoveries exclusively to gap-guided crawling—sub-tier vacuum suppliers (Edwards Vacuum, Leybold), assembly specialists (MacDermid Alpha), and international subsidiaries (ASMPT)—each traced to sectors flagged as degree anomalies.

Coverage estimation behavior

The Chao1-based coverage estimates warrant scrutiny. The estimator consistently overestimates coverage error relative to ground truth, peaking at 9.5 percentage points at iteration 3 before contracting to 5.3%. More consequentially, true coverage is only 10.3% (20/195 ground-truth companies), yet $\theta_r = 0.3$0 fluctuates between 8–20%, driven largely by singleton growth ($\theta_r = 0.3$1 reaches 669 vs.\ $\theta_r = 0.3$2 by iteration 5) as the crawler enters new web regions. Bootstrap 95% intervals on the population estimate ([565, 1127] at iteration 5) are wide, indicating substantial remaining uncertainty. The authors concede that Chao1 estimates coverage of the observed entity population—including non-ground-truth entities—rather than coverage of verified suppliers specifically, so the estimator functions best as a relative convergence signal rather than an absolute completeness measure.

A Michaelis–Menten fit to the discovery curve projects diminishing returns beyond roughly 20 iterations and provides an asymptotic population estimate independent of Chao1, though this projection extrapolates well beyond the five observed data points.

Relation quality and link-prediction validation

Type-consistency in the final KG is uneven: overall consistency is 85.7%, but supplies_to relations achieve only 6.5% type consistency (2 of 31 edges), versus 96.2% for located_in. This is a notable weakness given that supply relations are the graph's core value for supply-chain mapping; post-hoc filtering evidently does not fully compensate for zero-shot extraction errors on rare relation types.

The retroactive DistMult evaluation is candid about a null result: the exact-link confirmation rate is 0.0% at every iteration, attributed to the extreme sparsity of the supplies_to relation (31 edges total). However, entities involved in DistMult predictions gain new connections at elevated rates in early iterations (17.4% and 16.7% for iterations 1→2 and 2→3), falling to under 4% later. The paper accordingly positions link prediction as a supplementary signal only—an honest characterization supported by the data, but one that leaves open whether neural link prediction contributes materially at all.

Limitations and open questions

The paper acknowledges several constraints that bear directly on its conclusions. The curated ground truth is approximate—it may omit small firms and include acquired entities—and entity matching relies on case-insensitive string comparison, which likely depresses measured recall. The coverage estimator targets only the web-discoverable population, excluding firms with minimal web presence. Zero-shot extraction may hallucinate entities or miss implicit relations. Evaluation is restricted to a single sector at proof-of-concept scale (213 pages, 5 iterations, ~20 minutes wall-clock on a laptop); economy-wide scaling would require distributed infrastructure. Shared HTTP/LLM caches across baselines also mean crawl-ordering effects on extraction are not isolated. Key open questions include whether type constraints can be enforced within the prompt schema rather than post-hoc, whether richer GNN-based gap detection would outperform structural heuristics, and whether coverage estimation can be calibrated specifically to ground-truth-quality populations rather than observed entity pools.

Conclusion

This paper contributes a coherent iterative architecture that couples focused crawling, zero-shot LLM-based KG construction, and ecology-inspired coverage estimation for supplier discovery. Its strongest empirical claims are precision improvements under matched budgets, early peak recall, and near-zero marginal LLM cost; its most instructive negative results concern the weak absolute recall achievable against imperfect ground truth, Chao1's sensitivity to singleton inflation, and the failure of exact link confirmation for sparse relation types. The framework generalizes in principle to any domain where structured information is fragmented across the web, though the single-sector, small-scale evaluation leaves the breadth of that generalization untested.