Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Establish whether sample-based configuration rankings transfer to complete corpora

Determine whether a configuration ranked lower on stratified readability-assessment samples would outperform the selected configuration when evaluated on the complete corpora.

Background

The experimental configurations are ranked using stratified samples of at most 567 documents from each corpus, and only the highest-ranked configuration is subsequently evaluated on the complete corpora. Consequently, the study does not verify whether the sample-based ranking remains valid at full scale.

The mean Spearman correlation declines from 0.772 on the sampled datasets to 0.694 on the complete corpora. The authors therefore leave unresolved whether another configuration, ranked lower on the samples, would transfer better to the full datasets.

References

Mean $\rho$ falls from $0.772$ on the samples to $0.694$ on the complete corpora, and we cannot rule out that a configuration ranked lower on the samples would transfer better.

— Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting  (2609.24650 - Thieffry et al., 21 Sep 2026) in Section 6, Limitations, item 3

Our results therefore do not establish that three pages suffice for arbitrary hosts or for pages outside the observed crawls.

— Host-Aware Web Crawling for LLM Pretraining Data  (2610.13321 - Ostendorff, 9 Oct 2026) in Section Limitations, paragraph “Selection of pages and hosts”

A preceding development analysis of the same signal gave a median within-location rank correlation of $+0.003$ over 24 locations, and the two analyses are inconsistent in a way the available evidence does not resolve.

— Observation Quality in DSM-Assisted Aerial Localization: A Benchmark and Base-Anchored Uncertainty Calibration  (2610.18564 - Yin et al., 13 Oct 2026) in Section 6, subsection “Ranking and Calibration”