Establish whether sample-based configuration rankings transfer to complete corpora
Determine whether a configuration ranked lower on stratified readability-assessment samples would outperform the selected configuration when evaluated on the complete corpora.
References
Mean $\rho$ falls from $0.772$ on the samples to $0.694$ on the complete corpora, and we cannot rule out that a configuration ranked lower on the samples would transfer better.
— Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting
(2609.24650 - Thieffry et al., 21 Sep 2026) in Section 6, Limitations, item 3
Our results therefore do not establish that three pages suffice for arbitrary hosts or for pages outside the observed crawls.
— Host-Aware Web Crawling for LLM Pretraining Data
(2610.13321 - Ostendorff, 9 Oct 2026) in Section Limitations, paragraph “Selection of pages and hosts”
A preceding development analysis of the same signal gave a median within-location rank correlation of $+0.003$ over 24 locations, and the two analyses are inconsistent in a way the available evidence does not resolve.
— Observation Quality in DSM-Assisted Aerial Localization: A Benchmark and Base-Anchored Uncertainty Calibration
(2610.18564 - Yin et al., 13 Oct 2026) in Section 6, subsection “Ranking and Calibration”