Qwant Web Dataset: Dynamic Retrieval Benchmark

• Qwant Web Dataset is a dynamic benchmark that organizes Web queries, SERPs, and documents into monthly sub-collections.
• It simulates an evolving search environment by capturing temporal drift in query distributions and document collections.
• Relevance is determined via a debiased click-model that converts continuous click probabilities into discrete grades for standard evaluations.

Within LongEval-Retrieval, the Qwant Web Dataset is the Qwant-derived collection of Web queries, SERPs, documents, and click-based relevance assessments used to instantiate a dynamic test collection for continuous Web search evaluation. It is built from the production logs of Qwant, a privacy-preserving European Web search engine that primarily focuses on the French market, and it is organized as a sequence of monthly sub-collections rather than a single static Cranfield-style benchmark. The design goal is to simulate an evolving information system environment in which the document collection, the query distribution, and relevance all move continuously, while still supporting offline evaluation through fixed releases and standardized relevance files (Deveaud et al., 2023).

1. Source logs and temporal scope

The source data are drawn from Qwant’s production logs. These logs provide anonymized query text, the exact SERP that Qwant displayed for each query, and one click position per query if a click occurred. They do not contain user IDs, session chains, dwell time, or multi-click sessions. The logs are filtered to queries that Qwant itself answers rather than those forwarded to a third-party engine (Deveaud et al., 2023).

The temporal structure is central. Topic selection was performed in May 2022 on historical logs, and the released benchmark is composed of monthly sub-collections: June 2022 for training, July 2022 for short-term testing, and September 2022 for long-term testing. Each sub-collection covers roughly one calendar month of queries and clicks. This monthly organization operationalizes continuous retrieval evaluation by making temporal drift an explicit part of the benchmark rather than an uncontrolled nuisance variable.

The collection follows a dynamic test collection paradigm. Instead of fixing all benchmark components once, it fixes only the topic inventory and allows the observed queries, retrieved documents, and relevance estimates to vary by month. This permits direct comparison of system behavior across time steps under a common offline-evaluation protocol.

2. Monthly sub-collections, topics, and query selection

Each monthly sub-collection indexed by time stamp $t$ contains four components: a fixed set of top-level topics $T$, a shifting set of queries $Q_t$, a document corpus $D_t$, and soft relevance estimates for each $(q,d)\in Q_t \times D_t$. The topics are chosen once and reused across months; the queries and documents are refreshed from monthly logs (Deveaud et al., 2023).

The topic set $T$ consists of 28 French multi-word terms such as eau, voiture, loi, guerre, and eurovision. These topics were selected to be simultaneously popular, with at least $1\,000$ matching queries per month, stable according to recurrence in Google Trends and Qwant logs, and general and diverse. For a topic $t\in T$, the raw candidate query set is defined as

$$Q_t = \{ q \mid q \in \text{all Qwant-answered queries},\ t \subseteq_{\mathrm{str}} q \}.$$

Because $|Q_t|$ can reach tens of thousands, only the top-$T$0 most frequent queries per topic are retained. Queries with fewer than 10 logged impressions or clicks are removed, near-duplicates such as “anti virus” and “antivirus” are manually merged, and adult-content queries are discarded.

This construction fixes semantic scope at the topic level while allowing the realized query surface forms to drift over time. A plausible implication is that the benchmark is designed to measure both retrieval effectiveness and the persistence of that effectiveness under realistic query-distribution shift.

3. Document corpus construction and cleaning

For each monthly sub-collection, the document set $T$1 is the union of two sources. First, it includes every URL that appeared in a SERP for some $T$2 during month $T$3, together with the snapshot of page text extracted at crawl time. Second, it includes up to $T$4 random URLs per topic, sampled from the full Qwant index using an AND match on the tokens of the topic (Deveaud et al., 2023).

The random background component serves two stated purposes: ensuring a realistic prevalence of non-relevant pages and avoiding ranking-bias overfitting. This means the benchmark is not restricted to documents already surfaced by the production ranker. It therefore exposes systems to a broader candidate space than the observed click logs alone would provide.

All pages in $T$5 are cleaned through Qwant’s internal HTML-to-text pipeline, and explicit adult and spam filtering is applied. The resulting corpus is therefore neither raw HTML nor an unrestricted crawl. It is a processed Web corpus aligned with the SERP and click observations of the corresponding month.

4. Click-model-based relevance estimation

The relevance layer is derived from clicks, but raw clicks are not treated as direct judgments. The construction explicitly states that Qwant clicks are both noisy, because click is not equivalent to relevance, and biased, notably by position bias. Since Qwant logs only record one click per query, the full DBN of Chapelle and Zhang reduces to the original Cascade Model of Craswell et al.; the key estimated quantity is the attractiveness parameter $T$6 (Deveaud et al., 2023).

With maximum-likelihood estimation over all impressions $T$7 in which document $T$8 was shown for query $T$9 at or above the clicked rank, the paper gives

$Q_t$0

where $Q_t$1 is the set of all display events in month $Q_t$2 where $Q_t$3 appeared for query $Q_t$4 at rank less than or equal to the observed click rank, and $Q_t$5 is 1 if and only if $Q_t$6 was clicked in event $Q_t$7. These continuous $Q_t$8 values are used as soft relevance scores.

To support classical IR metrics that require discrete grades, $Q_t$9 is mapped into $D_t$0 by inverting the DCG-based mapping $D_t$1 from Chapelle et al. (2009):

$D_t$2

so that

$D_t$3

Each $D_t$4 pair therefore receives both a real-valued $D_t$5 and a discrete grade $D_t$6. A common misconception is that the collection consists of raw click labels; in fact, it consists of de-biased click-model estimates plus a discrete projection for standard evaluation.

5. French-English mirror, data formats, and access

Although the underlying traffic is predominantly French, the collection includes an English “mirror.” Every French query and every sentence of every French document is translated with CUBBITT, identified as the Charles University Transformer system. The pipeline first segments pages into sentences with spaCy’s French model, then uses FastText to detect truly French sentences, skipping sentences below a confidence threshold or of length 1 character. Sentences are truncated to at most 500 bytes, translated via REST calls to the LINDAT/CLARIAH-CZ CUBBITT API, and then reassembled into English documents (Deveaud et al., 2023).

The paper reports that translation quality is high for longer passages, while short queries can mistranslate; the example given is “cuisson gigot agneau” becoming “leg leg leg.” This directly qualifies the status of the English release: it is a translated mirror rather than a natively authored English Web-search collection. This also explains why the benchmark is suitable for cross-lingual and non-French experimentation, but not identical in linguistic properties across languages.

Each sub-collection is distributed in two standard layouts. In TREC format, the release contains a .docs file with TREC DOCNO and <TEXT>, and a .qrels file with qid docid grade where grades range from 0 to 2. In JSON/Anserini format, the release contains docs.jsonl with fields { "id":..., "url":..., "text_fr":..., "text_en":... }, queries.jsonl with { "id":..., "text_fr":..., "text_en":... }, and qrels.jsonl with { "qid":..., "docid":..., "alpha":..., "grade":... }. The June 2022 training sub-collection is publicly available on the LINDAT repository at http://hdl.handle.net/11234/1-5010, and the test collections for July 2022 and September 2022 were scheduled for release in April 2023 under a similar license. All user data are anonymized, and the click-model parameters are derived only from aggregated counts.

6. Composition, baseline results, and temporal-persistence evaluation

For the June 2022 training split, the paper reports a document corpus of $D_t$7 Web pages, 672 training queries, and 98 held-out queries. The total number of graded assessments is 9,656 for train and 1,420 for held-out, corresponding to approximately 14 grades per query on average. The training grade distribution is 73% non-relevant (grade 0), 21% relevant (grade 1), and 6% highly relevant (grade 2). Held-out queries are hidden during system development in order to measure robustness when moving from train to test (Deveaud et al., 2023).

Component	Value
Document corpus $D_t$8	1,570,734 Web pages
Train queries	672
Held-out queries	98
Graded assessments	9,656 train; 1,420 held-out
Average assessments per query	$D_t$9
Train grade distribution	73% grade 0; 21% grade 1; 6% grade 2

The baseline validation uses two off-the-shelf systems on both French and English sub-collections: Terrier with French stemming and stopwords on French and default BM25, and Anserini with Lucene+Java, no language-specific processing, and default BM25. Both systems return the top-1,000 documents per query. The reported metrics are

$(q,d)\in Q_t \times D_t$0

$(q,d)\in Q_t \times D_t$1

$(q,d)\in Q_t \times D_t$2

On the June 2022 training queries, the reported values are as follows.

System	MAP / P@10	nDCG / Recall@1000
Anserini (fr)	0.162 / 0.099	0.310 / 0.737
Terrier (fr)	0.177 / 0.111	0.331 / 0.770
Anserini (en)	0.137 / 0.089	0.277 / 0.677
Terrier (en)	0.146 / 0.094	0.290 / 0.706

On the held-out queries, the corresponding values are:

System	MAP / P@10	nDCG / Recall@1000
Anserini (fr)	0.168 / 0.111	0.325 / 0.757
Terrier (fr)	0.190 / 0.118	0.354 / 0.812
Anserini (en)	0.127 / 0.088	0.265 / 0.637
Terrier (en)	0.133 / 0.091	0.269 / 0.637

The English runs are reported as approximately 10% lower than the French runs, which the paper interprets as confirming that the machine translations are reasonably faithful to the original semantics. Similar performance on train and held-out queries is presented as evidence that the held-out queries are representative. These are empirical properties of the released benchmark rather than universal characteristics of translated retrieval collections.

Temporal persistence is evaluated by comparing system effectiveness across sub-collections. The benchmark explicitly motivates short-term persistence as $(q,d)\in Q_t \times D_t$3 and long-term persistence as $(q,d)\in Q_t \times D_t$4, and it defines the Relative nDCG Drop as

$(q,d)\in Q_t \times D_t$5

where $(q,d)\in Q_t \times D_t$6 is a retrieval system, $(q,d)\in Q_t \times D_t$7 is the query set at time $(q,d)\in Q_t \times D_t$8, and $(q,d)\in Q_t \times D_t$9 is a later test month. A small negative $T$0 indicates robustness to temporal drifts. This places the Qwant Web Dataset in a specific methodological niche: it is not only a Web retrieval benchmark, but a benchmark for studying when effectiveness degrades as the Web and user-query distributions evolve.