ClueWeb-Reco: Web Recommendation Benchmark

• ClueWeb-Reco is a synthetic webpage recommendation benchmark built from anonymized U.S. browsing histories, mapped to the ClueWeb22 corpus for zero-shot evaluation.
• It features a realistic setting with an 87-million-item candidate space, high sparsity, and diverse user browsing sequences reflecting real-world behavior.
• The dataset uses leave-one-out sequential splits and ranking metrics like Recall and NDCG to assess models while ensuring privacy through one-to-one synthetic mapping.

Searching arXiv for the cited works and closely related ClueWeb-based recommendation context. ClueWeb-Reco is the webpage recommendation component of ORBIT, the Open Recommendation Benchmark for Reproducible Research with Hidden Tests. It is a synthetic but behavior-preserving dataset built from real, user-consented U.S. browsing histories, aligned to 87 million public webpages from the ClueWeb22-B English corpus, and reserved as the hidden evaluation task of the ORBIT leaderboard rather than as an open training set. Its design targets a realistic recommendation setting with a very large candidate space, high sparsity, diverse domains, and realistic browsing sequences, while testing whether models trained on public recommendation datasets can generalize zero-shot to a completely disjoint open-web domain (He et al., 30 Oct 2025).

1. Role within the ORBIT benchmark

ClueWeb-Reco occupies a distinct position inside ORBIT. ORBIT’s public benchmark evaluates models on five standard datasets—MovieLens-1M and four Amazon categories—with open data, reproducible splits, and transparent settings. By contrast, ClueWeb-Reco is used only as a hidden evaluation task on the leaderboard, with models trained on public datasets and then evaluated zero-shot on webpage recommendation sessions whose items are disjoint from the training domain (He et al., 30 Oct 2025).

The rationale is to address several limitations of established recommender-system corpora. Many widely used datasets are built from ratings or reviews rather than browsing, and therefore capture sparse and potentially biased user behaviors. Many also use relatively small item vocabularies, typically from thousands to hundreds of thousands of items, rather than a web-scale candidate pool. ClueWeb-Reco is intended to be closer to browser-style recommendation—operationally, “what web page should this user see next?”—while preserving privacy through synthetic mapping into a public research corpus (He et al., 30 Oct 2025).

A common misconception is that the descriptor “synthetic” implies unconstrained data generation. In ClueWeb-Reco, synthetic refers instead to a privacy-preserving remapping of real browsing sequences into semantically similar ClueWeb22 documents, with exact-hit avoidance and one-to-one mapping, so that no real URL appears in the released sequences (He et al., 30 Oct 2025).

2. Data collection, quality control, and privacy-preserving construction

The original source consists of user-consented browsing histories collected via Amazon Mechanical Turk and Prolific.co under an exempt IRB protocol at Carnegie Mellon University, study ID STUDY2025_00000079. Participants were at least 18 years old, located in the U.S., and explicitly consented to share browsing histories. Submission occurred through two modalities: “Manual Fill,” in which subjects pasted approximately 15 URL–timestamp pairs, and “Edge Browser Export,” in which subjects exported Edge history and pasted 20–30 records. Participants were instructed not to submit URLs with personal information, to submit English-language pages, to represent actual browsing within the past year, and to prefer entertainment-related content when possible (He et al., 30 Oct 2025).

Online quality control filtered each URL–timestamp pair. A pair was accepted only if the URL was syntactically valid and within the past year, publicly accessible, not a landing page or generic search engine results page, in English, and passed anti-scam checks. A submission was stored only if it contained at least 10 valid pairs. Offline quality control then removed scam submissions, inappropriate content, and non-informative data; URLs were scraped and further filtered by language and informativeness, and sequences with fewer than 5 records were removed. The collection statistics reported for the raw-to-filtered pipeline are 1,747 consented subjects, 2,682 raw submissions passing online QC, and 41,760 URL–timestamp pairs, which after offline QC and filtering became 1,024 sessions and 12,282 browsing records, with average sequence length approximately 11.99 and maximum length 137 (He et al., 30 Oct 2025).

Privacy protection is implemented through soft matching into ClueWeb22-B EN. For ClueWeb22-B EN pages, title plus cleaned content is encoded; for collected URLs, scraped page content is encoded; the encoder is MiniCPM-Embedding-Light. An approximate nearest-neighbor index over the 87 million ClueWeb22-B EN pages is built using DiskANN. For each collected URL, the system retrieves the nearest ClueWeb22-B EN document in embedding space and uses the Top-1 candidate by default. About 11.07% of collected URLs are exactly present in ClueWeb22 after normalization; in those cases, the exact match is discarded and Top-2 is used instead. This enforces two properties: all released sequences are synthetic, and mapping is one-to-one, so that the same collected URL maps to the same ClueWeb ID while different collected URLs map to different ClueWeb IDs (He et al., 30 Oct 2025).

The released data contains only ClueWeb22 document IDs in sequences. No original URLs, timestamps, or textual content from user histories are released. To access actual webpage titles, URLs, or text, a researcher must separately obtain ClueWeb22 under its research-only license. Participant IDs are anonymized and decoupled from platform worker IDs after compensation. The design is described as following the privacy-preserving precedent of MSMARCO and TREC CAsT, where real behavior is transformed into sequences over public documents (He et al., 30 Oct 2025).

3. Sequential recommendation task and evaluation protocol

ClueWeb-Reco defines a sequential webpage recommendation task, specifically next-item prediction. For a session sequence of item IDs,

$s = (i_1, i_2, \dots, i_T),$

the task is to predict $i_T$ given the prefix $(i_1, \dots, i_{T-1})$ in the test setting. The formulation is therefore session-based sequential ranking rather than pointwise rating prediction or regression (He et al., 30 Oct 2025).

ORBIT applies the same leave-one-out sequential split across all datasets. For a sequence of length $n$, training uses the first $n-2$ items, validation uses the first $n-2$ items as input with target $i_{n-1}$, and test uses the first $n-1$ items as input with target $i_n$. For ClueWeb-Reco, Table 2 reports 1,024 sessions, with train count marked as N/A and both validation and test containing 1,024 targets. In practice, models are not trained directly on ClueWeb-Reco; they are trained on public datasets such as Amazon Books and evaluated zero-shot on ClueWeb-Reco validation and test (He et al., 30 Oct 2025).

The observable input in ClueWeb-Reco evaluation is a sequence of ClueWeb22 document IDs. In the LLM-QueryGen baseline, this history is verbalized as the titles of those pages, which are obtainable from ClueWeb22. The target is a single next-webpage ID per session. Evaluation is ranking-based: models must produce a ranked list over the full item pool, and performance is computed from the single ground-truth item using Recall@$K$ and NDCG@$i_T$0, with $i_T$1 (He et al., 30 Oct 2025).

With one relevant item per query session, Recall@$i_T$2 reduces to HitRate@$i_T$3: $i_T$4 For NDCG@$i_T$5, the standard discounted formulation is used. Because there is only one relevant item, NDCG@$i_T$6 becomes the average of the reciprocal log-discount at the rank where the ground-truth item appears, truncated at $i_T$7 (He et al., 30 Oct 2025).

4. Dataset characteristics and benchmark difficulty

ClueWeb-Reco is small in interaction count and extreme in item-space size. The processed dataset statistics are 1,024 sessions, 87,208,655 items, 12,282 interactions, and 99.99999% sparsity. The average sequence length is approximately 11.99, with a long tail up to 137. The dataset therefore combines short, cold-start-like sessions with longer, warmer sessions inside an item universe that is orders of magnitude larger than those of MovieLens-1M or the Amazon categories used in ORBIT’s public benchmark (He et al., 30 Oct 2025).

The reported domain-distribution analysis shows that the top domains in raw collected data, including YouTube as the most frequent domain, are closely mirrored after soft matching to ClueWeb22. The paper interprets this as evidence that soft matching preserves domain-level behavior patterns. An example session reported in Table 7 moves across Amazon MacBook pages, medical consultation sites, recipe pages, YouTube music videos, used MacBooks, and eBay cancellation help, with the target being another eBay page about order cancellation. This illustrates rapid shifts in intent and the need to model both short-term and long-term interests in a diverse web environment (He et al., 30 Oct 2025).

The benchmark results emphasize the difficulty of the task. Because all ClueWeb-Reco items are unseen during training, ORBIT evaluates only content-based baselines on this hidden test. TASTE, trained on Amazon Books, obtains Recall@10 $i_T$8, NDCG@10 $i_T$9, Recall@100 $(i_1, \dots, i_{T-1})$0, and NDCG@100 $(i_1, \dots, i_{T-1})$1. HLLM, also trained on Amazon Books, improves to Recall@10 $(i_1, \dots, i_{T-1})$2, NDCG@10 $(i_1, \dots, i_{T-1})$3, Recall@100 $(i_1, \dots, i_{T-1})$4, and NDCG@100 $(i_1, \dots, i_{T-1})$5. Among the prompted LLM retrieval baselines, DeepSeek-V3-QueryGen is reported as best overall across all cutoffs, with Recall@10 $(i_1, \dots, i_{T-1})$6, NDCG@10 $(i_1, \dots, i_{T-1})$7, Recall@100 $(i_1, \dots, i_{T-1})$8, and NDCG@100 $(i_1, \dots, i_{T-1})$9 (He et al., 30 Oct 2025).

These results are presented as evidence for two simultaneous points. First, existing recommendation methods have clear limitations in large-scale webpage recommendation: even strong content-based models achieve very low absolute metrics. Second, LLM integration is promising: several LLM-QueryGen variants match or exceed HLLM, and the reported case studies show that small differences in query wording can substantially alter retrieval quality. The paper also notes that ID-based sequential models are not evaluated on ClueWeb-Reco because they fundamentally cannot handle new IDs not seen in training (He et al., 30 Oct 2025).

5. Hidden-test mechanism, access model, and reproducibility

ClueWeb-Reco is the hidden part of the ORBIT leaderboard hosted at https://www.open-reco-bench.ai, and the dataset card is hosted at https://huggingface.co/datasets/cx-cmu/ClueWeb-Reco. Publicly available artifacts include the sequences of ClueWeb22 document IDs with split labels, aggregate statistics such as counts, domain distributions, sparsity, and session-length histograms, as well as code and baseline configurations. Not public or restricted are the original user URLs and timestamps, the original browsing histories, and the ClueWeb22 textual content unless separately licensed. Most importantly, the test labels for leaderboard evaluation remain hidden; participants submit predictions, and evaluation is performed server-side (He et al., 30 Oct 2025).

The hidden-test mechanism is meant to reduce overfitting and gaming. Hidden labels prevent direct tuning on test outcomes. Because the dataset is derived from real, private histories and no public version of the original logs is released, external data leakage is described as unlikely. The combination of an 87-million-item space and complete disjointness from the public training sets further reduces the possibility of memorization or reverse engineering (He et al., 30 Oct 2025).

The ClueWeb-Reco dataset itself is released under an MIT license, while use of titles, URLs, and full text requires a separate ClueWeb22 license from CMU. ORBIT’s codebase is available at https://github.com/cxcscmu/RecSys-Benchmark and includes data processing scripts, model implementations for baselines, training and evaluation pipelines for public datasets, and scripts for zero-shot ClueWeb-Reco evaluation. Standardization across ORBIT is achieved through common sequential leave-one-out splits, full ranking over the entire item pool, and Recall/NDCG@$n$0 metrics (He et al., 30 Oct 2025).

The ethical frame is explicit. The collection uses informed consent, IRB approval, and synthetic mapping to a public corpus instead of release of private browsing logs. Worker compensation is reported as approximately \$n$10.4 per valid submission (He et al., 30 Oct 2025).

6. Position in the broader ClueWeb research landscape

Relative to standard recommendation benchmarks such as MovieLens, Amazon, or Yelp, ClueWeb-Reco is distinguished by three properties stated in ORBIT: it is built from real, user-consented browsing histories, it maps those histories to a public web corpus through semantic soft matching, and it evaluates zero-shot generalization over an 87-million-item candidate space as a hidden test rather than a tuning set (He et al., 30 Oct 2025). This design situates it at the intersection of recommender systems, web retrieval, privacy-preserving data release, and evaluation methodology.

Subsequent work makes this positioning explicit. “WebRec: Enhancing LLM-based Recommendations with Attention-guided RAG from Web” formulates a practical ClueWeb-style recommendation system in which a recommendation prompt is converted into web queries, top-$n$2 pages or snippets are retrieved, and an LLM equipped with an MP-Head uses the noisy long-context evidence to choose the next item from a candidate set. WebRec is motivated by the gap between structured recommendation metadata and heterogeneous open-web evidence, and it states that its mechanisms apply directly to a ClueWeb-style setting by replacing live web search with a ClueWeb index (Zhao et al., 18 Nov 2025). A plausible implication is that ClueWeb-Reco can function not only as a benchmark for zero-shot transfer, but also as a target environment for retrieval-augmented LLM recommender architectures.

An earlier antecedent appears in “Ranking Triples using Entity Links in a Large Web Crawl - The Chicory Triple Scorer at WSDM Cup 2017,” which used ClueWeb12 annotated with FACC1 entity links to score person–profession and person–nationality triples through document-level co-occurrence counts and simple fusion with Wikipedia evidence (Dorssers et al., 2017). That task is not sequential recommendation, but it demonstrates an earlier ClueWeb-based paradigm in which a large web crawl supplies relevance signals through structured mappings over public web documents. This suggests continuity between entity-centric relevance estimation over ClueWeb12 and session-based webpage recommendation over ClueWeb22.

Within ORBIT, the expected future use of ClueWeb-Reco is as a stress test and generalization benchmark for models trained on curated product domains and then transferred to open-web recommendation. The authors also indicate plans to expand ORBIT with additional public datasets and to extend the ClueWeb-Reco pipeline toward larger supervised datasets while preserving privacy (He et al., 30 Oct 2025).