Evaluate combined exploration and sample-guided depth allocation

Determine whether combining centrality-based exploration for discovering new URLs and hosts with sample-guided depth allocation improves crawl quality or discovery outcomes.

Background

The crawl simulations allocate a fixed budget among already known pages and therefore do not measure the discovery benefits of following links from central or otherwise strategically selected hosts. The paper distinguishes this discovery objective from quality-oriented depth allocation and proposes combining the two mechanisms. The effectiveness of that combined policy is left unresolved.

References

A crawler could combine centrality-based exploration with sample-guided depth allocation, but our experiments do not evaluate this combination.

— Host-Aware Web Crawling for LLM Pretraining Data  (2610.13321 - Ostendorff, 9 Oct 2026) in Section Discussion, paragraph “Quality and discovery are different objectives”