Determine the downstream-model value of recovering hidden high-quality pages

Quantify how much the estimated high-quality URL budget on under-crawled hosts improves trained-model performance by conducting a targeted re-crawl of top hidden-gem hosts followed by a downstream ablation.

Background

The analysis estimates that tens of millions of known but uncrawled URLs on under-crawled hosts may be high quality, using each host’s observed quality density to estimate the quality of its uncrawled URLs. This estimate does not establish whether recovering those URLs would materially improve language-model training. The paper therefore identifies targeted re-crawling and downstream ablation as the unresolved way to measure the value of the missed crawl budget.

References

Recovering the hidden gems. RQ4 identifies tens of millions of known high-quality URLs on under-crawled hosts. A targeted re-crawl of the top gem hosts, followed by a downstream ablation, would quantify how much the missed budget is worth in trained-model terms.

— Host-Aware Web Crawling for LLM Pretraining Data  (2610.13321 - Ostendorff, 9 Oct 2026) in Section Future Work, subsection “Recovering the hidden gems”