Assess whether centrality-assisted sampling improves crawl yield

Determine whether using harmonic centrality or PageRank to group hosts for sampling improves the high-quality-page yield of host-level crawl allocation.

Background

The experiments find that ordering hosts directly by graph centrality provides limited or negative gains relative to random ordering. The paper nevertheless notes that centrality might be useful for grouping hosts before applying a sampling strategy, which is a different use from ranking hosts by presumed content quality. That proposed combination was not tested, so its effect on crawl yield remains unresolved.

References

Lastly, centrality could still be used to group hosts for sampling, although we do not evaluate whether this improves crawl yield.

— Host-Aware Web Crawling for LLM Pretraining Data  (2610.13321 - Ostendorff, 9 Oct 2026) in Section Discussion, paragraph “Explaining earlier findings on graph centrality”