The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Salience

Abstract: A social highlighter's most useful signal -- which passages a crowd of readers marks -- exists only for documents people have already read. Can the aggregate crowd salience of a document be predicted from its text before its marks accumulate? Prior work on this data found that zero-shot LLMs recover highlight locations worse than a trivial lead (position) baseline, so we ask whether a model trained on the highlight corpus can beat that baseline. Using a pre-registered ladder of models and a by-document cluster bootstrap, we find a small but robust edge: a logistic ranker over sentence embeddings and positional/contextual features beats the lead baseline by +0.044 average precision (95% CI [+0.029, +0.058]; clears a pre-registered margin delta=0.03 in 97% of resamples, and stable across pipeline re-runs). Two unsupervised extractive baselines (centroid, LexRank-style centrality) lose to lead, and the trained model beats them by +0.108, so the edge is not recovered by generic unsupervised proxies -- it reflects learning from real reader marks. In product terms, precision@3 rises from 0.25 to 0.39 (+55% relative) and the model beats lead on 69% of documents. An ablation attributes the edge to the raw embedding (+0.014) and training augmentation (+0.010), each with a positive CI. The edge is not a temporal-generalization failure, and we find no evidence that content drift or near-duplicate leakage explains it. A standardized regression shows the advantage is governed mainly by document popularity (lower popularity, larger edge) and by label reliability. It nearly vanishes only on the most popular content; there it is the lead baseline that strengthens, not the model that weakens. Because our evaluation conditions on documents that eventually accumulated readers, these results are a retrospective cold-start simulation.

• The paper introduces a supervised logistic ranker that achieves a +0.044 AP gain over lead baselines for predicting future crowd highlights.
• Methodologically, it integrates sentence embeddings, contextual signals, and positional features, validated through robust bootstrap evaluations and ablation studies.
• Empirical results reveal a significant product lift with a 55% improvement in top-3 precision, especially in less-popular, long-tail web documents.

Cold-Start Prediction of Crowd Highlight Salience: Empirical Evidence Beyond Lead Baselines

Problem Formulation and Motivations

The paper "The Long Tail, Not the Front Page: Cold-Start Prediction of Crowd Highlight Salience" (2606.11654) formalizes the cold-start salience prediction problem for social document highlighting platforms. Unlike prior extractive summarization and salience modeling, which typically reference gold-standard summaries or small annotator panels, this work addresses the aggregate highlight patterns of real reader crowds, which are essential for power-user features such as popular highlighting overlays and socially-informed discovery. The cold-start scenario—predicting which spans a future collective will highlight before any marks exist—poses unique challenges and has clear product relevance.

Prior findings demonstrated that zero-shot LMs underperform trivial positional (lead) baselines at this task (Nakayashiki et al., 9 Jun 2026), and that customization to individual user profiles does not improve aggregate salience order (Nakayashiki et al., 8 Jun 2026). Consequently, the central hypothesis here is whether supervised learning on historical crowd highlight corpora can produce a model that beats strong position-based baselines in predicting “future” highlights on documents lacking marks.

Methodology and Experimental Framework

The study leverages largescale data from Glasp, a web annotation platform, focusing on documents with established crowd signals. Documents are filtered to retain only those with at least 20 lifetime unique highlighters; for each, highlights are sentence-anchored, forming a highly granular binary label for top 15% crowd-salience sentences. The downstream task is formulated as sentence ranking, with effectiveness evaluated primarily by average precision (AP) relative to the lead (position) order, using a by-document cluster bootstrap to robustly estimate confidence intervals.

A pre-registered model ladder explores increasingly expressive feature sets:

• M0: Positional and lexical features (normalized position, TF-IDF, log-length),
• M1: Pre-trained large sentence embeddings plus position,
• M2: Contextual signals (centroid and degree-centrality semantics, position, surface cues),
• M3: Combined representation, with data augmentation via inclusion of moderately popular training-only documents.

Baseline comparisons include random, lead (positional), and classic unsupervised extractive strategies (centroid similarity, LexRank-style centrality).

Main Results and Empirical Robustness

The principal finding is that a logistic ranker atop sentence embeddings and contextual/positional features (M3) exhibits a statistically robust advantage over the lead baseline:

Figure 1: The trained model's edge over lead grows with representation and crosses the pre-registered bar ($\delta = 0.03$).

The numerical headline is a mean $+0.044$ AP gain over lead ($95\%$ CI $[+0.029, +0.058]$), clearing the critical margin $\delta=0.03$ in $97\%$ of bootstrap samples. Notably, this advantage is not recoverable by unsupervised extractive approaches (centroid, LexRank), both of which underperform the lead baseline by $-0.065$ AP. This demonstrates that the signal is attributable not to shallow semantic centrality but to supervised learning from actual reader marks.

In product-relevant metrics, the top-3 precision rises from 0.254 (lead) to 0.394 (M3), a $+0.14$ absolute (or $+55\%$ relative) lift, and M3 dominates per-document AP over lead in 69% of documents, further underscoring practical benefit.

Ablation studies attribute the improvement to both direct sentence embedding features ($+0.014$) and data augmentation ($+0.044$0), with both factors showing distinct, bootstrap-supported contributions.

Error Analysis and Covariate Decomposition

A comprehensive set of post hoc analyses, including standardized regression and stratified grid breakdowns, localize the gain. The model’s edge over lead is largest in less-popular documents ("the long tail") and in regions where ground-truth labels are thicker (more reliable):

Figure 2: Model AP is stable across popularity × recency; the lead baseline improves dramatically on popular, recent documents, shrinking the advantage.

A fitted OLS shows the AP improvement is inversely correlated with document popularity ($+0.044$1 standardized coefficient), and positively with label thickness (number of co-readers; $+0.044$2), with negligible and non-robust effects from recency or length. The lead baseline’s own AP increases sharply on the most popular, recent documents—those where crowd highlights cluster at the start—explaining the vanishing model-to-lead gap there. The model itself does not degrade on these documents.

Robustness Checks

Stability is validated across overlapping pipeline re-executions and under near-duplicate removal, with the main effect actually increasing after deduplication. Controls for temporal drift (train/test split by document recency) reveal no shift, attributing observed performance variation to population composition rather than model generalization failure.

Figure 3: (a) Advantage over lead is constant across pipeline reruns, rises after removing near-duplicates. (b) Regression shows only popularity (−0.32) and label thickness (+0.22) as robust drivers.

Additional considerations are addressed: anchoring noise (from highlight-to-sentence alignment) is shared across all models; content drift due to fetching current article bodies is not explanatory; survivorship filtering (evaluating only on documents that eventually drew crowds) means the observed regime is a retrospective simulation, not prospective true zero.

Limitations and Theoretical Implications

Major caveats are acknowledged. First, zero-reader true cold-start is not directly evaluated due to survivorship bias—thin-label documents are the most difficult to measure cleanly. Second, supervised alternatives trained against abstracts or expert summaries are not explored, and the implemented model is limited to logistic rankers over fixed representations; stronger architectures might raise the observed ceiling.

Population-wise, fetchable documents are biased toward short and popular web articles due to accessibility constraints. The specifics of label definition (binary top-15%) introduce some arbitrariness, and further robustness across label granularities is left for future work.

Theoretically, this work delineates the boundary where shallow positional baselines transition from strong proxies to confounders requiring correction via supervised learning. The strong product lift in less-popular documents (long tail) contrasts with the front-page regime, where lead suffices.

Broader Impacts and Future Directions

For social/highlighting platforms, these findings endorse the prospect of automating cold-start highlight overlays in long-tail contexts, where organic crowd signals are sparser and temporal delays more acute. For extremely popular content, product utility is lower, as the lead order aligns with crowd highlights and organic signal accumulates quickly.

On the methodological front, the study emphasizes the necessity for resampling-robust effect estimates and cautions against over-interpretation of mechanistic hypotheses untested by knock-out or controlled experiments.

Future work should extend to fine-tuned transformers, alternate supervision signals (e.g., summary-based salience), true zero-reader validation populations, and robustification to content drift and anchoring artifacts.

Conclusion

This paper provides the first robust evidence that aggregate crowd highlight salience is text-predictable in the cold-start regime for web documents, with a trained model surpassing the strong lead baseline by $+0.044$3 AP and $+0.044$4 precision@3 in realistic, product-facing metrics. The gain is exclusively present in less-popular documents, reinforcing that the “long tail” is both where cold-start prediction is most necessary and most achievable, while “front-page” content remains dominated by positional priors. The work establishes a new baseline for product and methodological rigor in social salience prediction research.