Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems

Abstract: Evaluating first-stage retrievers in large-scale production RAG requires a benchmark that pairs a large-scale corpus with a large set of agent-reformulated search queries based on real user queries and their conversation threads, and that labels many relevant documents per query. No existing public benchmark evaluates this setting: large-scale collections typically provide only a small number of evaluation queries, whereas benchmarks with many queries generally contain only millions of documents. Moreover, most benchmarks assess human-written queries, while the first-stage retrievers in agentic RAG pipelines serve machine-written reformulations whose distribution differs from human search behavior. To overcome these evaluation gaps, we introduce Q2D-Web (Query2Doc-Web), a large-scale agentic retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten languages, reformulated from real-world user queries in production systems. Q2D-Web provides three sets of fixed relevance judgments: agent citations, production rankings, and a combined set that unions both signals and adds LLM-based judgments of unlabeled pooled documents to reduce false negatives. We benchmark 13 retrievers including lexical, dense, and late-interaction models and find that their relative ordering is largely insensitive to the choice of judgment set, while diverging substantially across topical domains, query languages, and query types. To enable fast evaluation, we also study subcorpus sampling as an approximation to full-corpus evaluations. Retaining a third of the corpus, selected by reciprocal rank fusion over pooled retriever runs, preserves the full-corpus model ranking under the combined judgments while raising absolute Recall@1000 only by 3 to 7 points. The public leaderboard is accessible under: https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard

• The paper introduces Q2D-Web, a benchmark evaluating retrieval in agentic retrival-augmented generation (RAG) systems using a large corpus of approximately 190 million web documents and 69,721 agent-reformulated queries in ten languages.
• Q2D-Web improves upon conventional benchmarks by including large query populations and machine-generated queries, typical of agentic multi-step searches, and employs Recall@1000 as its primary metric
• Q2D-Web identifies diverse errors in retrieval performance, highlighting that models performing well in aggregate recall may not necessarily recover many distinct relevant documents.

Q2D-Web addresses a specific evaluation mismatch in agentic retrieval-augmented generation (RAG): deployed systems do not retrieve directly from short, human-written benchmark queries, but from machine-generated reformulations produced within multi-step conversational searches. The benchmark therefore evaluates the first-stage retriever under the conditions that determine which evidence is available to downstream rerankers and agents. Its central resource is a corpus of approximately 190 million web documents paired with 69,721 agent-reformulated queries in ten languages, derived from nine months of privacy-filtered production traffic (2609.08887).

Evaluation problem and benchmark position

First-stage retrieval is conventionally assessed using benchmarks such as MS MARCO, BEIR, and TREC Deep Learning. These resources provide valuable relevance judgments, but they do not jointly provide production-scale corpora, large query populations, and machine-generated queries characteristic of agentic search. MS MARCO Web Search contains approximately 100 million documents but only 9,374 test queries and one click-derived positive per query. By contrast, Q2D-Web contains roughly seven times as many test queries and an average of 99.6 positive documents per query in its combined judgment set.

The distinction between human and agent-generated queries is methodologically consequential. Each production search begins with a user message, but the agent generates one primary query and potentially several supporting queries using the conversation history and previously retrieved results. Supporting queries can alter both the surface form and the set of documents satisfying the information need. Since retrieval performance is sensitive to formulation—even semantically preserving query variations can reduce effectiveness by approximately 20% (2609.08887)—benchmarks based exclusively on manually authored queries may misestimate the behavior of production retrievers.

Q2D-Web consequently evaluates retrieval as candidate generation rather than final ranking. Its primary metric is Recall@1000, reflecting the requirement that a first-stage retriever expose a broad evidence pool to subsequent rerankers. Recall@100 and nDCG@10 are also reported to characterize shallower retrieval quality. This metric choice produces a substantive distinction from conventional web-ranking evaluations: a model can be strongest at recovering relevant documents overall while another is better at placing those documents near the top.

Construction of queries and corpus

The query set comprises 12,365 primary queries and 57,356 supporting queries, with supporting queries accounting for 82.3% of the dataset. The queries were sampled monthly while matching production distributions over topic and language. Exact duplicates, very short queries, filtering-operator queries, and queries flagged for PII were removed. The ten most frequent languages were retained.

The paper emphasizes that these queries are not merely paraphrases of user input. A primary query typically restates the user’s intent in a more retrieval-oriented form, whereas supporting queries may decompose the information need, seek background information, resolve adjacent entities, or conduct additional retrieval hops. The examples include queries whose formulation depends on prior conversational context and support queries that alternate between languages. This structure directly tests whether retrieval models trained on conventional human-written query-document pairs generalize to the distribution generated by agents.

The corpus is constructed by retrieving the top 5,000 documents for every dataset query using a production retrieval system, taking the union, and deduplicating documents with MinHash–LSH at a token 5-gram Jaccard threshold of 0.975. The resulting collection contains approximately 190 million canonical documents, with the longest document retained within each near-duplicate cluster. The mean document length is approximately 13,169 characters.

This is a deliberately nonrandom corpus. Every document was considered plausible for at least one benchmark query by a production retrieval stack, and documents retrieved for one reformulation can become hard distractors for related reformulations. The benchmark therefore concentrates evaluation on discriminating relevant evidence from semantically similar, topically related, temporally mismatched, or otherwise plausible web content rather than from uniformly sampled irrelevant pages.

Figure 1: Q2D-Web construction pipeline from production query sampling through corpus assembly, multi-source relevance annotation, and subcorpus selection.

Multi-source relevance judgments

Q2D-Web provides three relevance-judgment sets over the same query-corpus pair: Citation, Web Ranking, and Combined+LLM-Judged. This design recognizes that no single production signal provides complete relevance supervision.

Citation labels a document as relevant when the agent cited it in its final response. This signal is closely connected to downstream RAG use, but it is intentionally incomplete. Agents stop citing once a claim is supported, so redundant but relevant documents remain unlabeled. Citation judgments are therefore high-precision and low-recall.

Web Ranking labels up to the top 50 results per query from a production system using BM25 and dense first-stage retrieval followed by cross-encoder reranking. This pass recovers relevant documents that citation behavior omits, but it inherits the biases of that ranking stack and may differ from citation labels because it is computed over a later corpus snapshot.

The Combined+LLM-Judged set unions the first two sources and adds judgments for previously unlabeled candidates. Nine lexical, dense, and late-interaction retrievers released before 1 January 2025 generate a reciprocal-rank-fused pool of up to 500 previously unjudged documents per query. DeepSeek-V4-Flash then assigns binary relevance labels using a strict constraint-matching prompt. The temporal separation between pooling models and evaluated neural retrievers reduces direct benchmark-construction overlap, but it does not eliminate annotation bias. The LLM judge may favor fluent or long-form documents, and all three sources are conditioned on their respective retrieval or generation systems.

The use of multiple judgment sets allows the authors to test whether model comparisons are artifacts of a particular labeling source. The principal result is that relative model ordering is largely stable across Citation, Web Ranking, and Combined judgments. This stability supports the robustness of the aggregate comparisons, although it should not be interpreted as evidence that the judgment sets are interchangeable: absolute scores differ, and the Citation set necessarily measures downstream citation behavior more narrowly than topical or evidential relevance.

Subcorpus sampling at web scale

Full-corpus evaluation is computationally prohibitive. A full pass with the 4-billion-parameter pplx-embed-v1 model requires 4,608 H200 GPU-hours, while the 300-million-parameter EmbeddingGemma model requires nearly 200 H200 GPU-hours. Q2D-Web therefore evaluates whether a carefully selected subcorpus can preserve full-corpus conclusions.

All positively judged documents are retained. The remaining documents are selected as distractors using depth-based pooling, RRF over pooled retriever runs, or uniform random sampling. RRF performs best at the principal operating point. Retaining approximately 31.7% of the corpus preserves the full-corpus model ordering with Kendall’s $\tau_b = 1.00$. At this size, RRF increases mean Recall@1000 by 5.1 percentage points relative to full-corpus evaluation, compared with 11.1 points for uniform sampling. Depth pooling achieves a smaller 3.9-point discrepancy only at a larger retained fraction of 43.4%.

The selected RRF configuration, using $k=1000$, is therefore an efficient approximation rather than an unbiased replacement for full-corpus evaluation. Increasing the pool to $k=2000$ reduces the recall gap to 0.7 points, but requires retaining 52.9% of the corpus. For pplx-embed-v1-4b, the adopted subcorpus reduces computation from 4,608 to approximately 1,500 H200 GPU-hours; EmbeddingGemma falls below 70 GPU-hours. The paper reports that evaluation can consequently be performed in approximately eight hours on a single 8×H200 node instead of a multi-node cluster.

The subcorpus analysis includes a temporal holdout: pre-2025 models construct the pool, while the evaluated neural models are released after that cutoff. This is important because pooling can favor participating systems. Nevertheless, the guarantee established by the experiments is empirical and conditional on the evaluated model families, metrics, judgment set, and sampling pool. A new retriever with substantially different behavior could expose omitted documents and alter the approximation.

Benchmark results across retriever families

The authors evaluate thirteen retrievers: BM25, ten dense encoders, and two late-interaction models. Neural systems range from 300 million to 8 billion parameters. Documents are truncated to their first 512 tokens for indexing, a major cost constraint given the corpus scale. The paper explicitly declines to claim that the reported scores are unbiased estimates of full-document effectiveness; longer-input spot checks did not reorder the models, but most document content remains unseen under the main protocol.

Under Combined+LLM-Judged relevance on the full corpus, pplx-embed-v1-4b achieves the highest Recall@1000 at 69.11%, followed by Nemotron-3-Embed-8B at 68.58%. EmbeddingGemma-300M reaches 65.45%, while Qwen3-Embedding-8B reaches 64.53%. BM25 has the lowest aggregate Recall@1000 at 44.77%. The same winning models remain winners under subcorpus evaluation, although subsampling raises absolute Recall@1000 by approximately 3–7 points.

Retriever	Full Recall@1000, Combined	Subcorpus Recall@1000, Combined	Interpretation
pplx-embed-v1-4b	69.11	74.07	Highest overall recall
Nemotron-3-Embed-8B	68.58	72.79	Strongest shallow ranking
EmbeddingGemma-300M	65.45	70.10	Strong sub-1B baseline
Qwen3-Embedding-8B	64.53	69.61	Monotonic scaling within family
pplx-embed-v1-0.6b	67.02	71.52	Strongest sub-1B dense model
mLateOn	60.98	65.62	Late interaction improves over mDenseOn
BM25-tantivy	44.77	49.85	Lowest aggregate recall

The metric-dependent leadership is particularly important. pplx-embed-v1-4b leads Recall@1000, but Nemotron-3-Embed-8B leads Recall@100 and nDCG@10 under Combined judgments. Nemotron therefore places a larger fraction of its recovered positives near the top, whereas pplx-embed-v1-4b recovers more positives across the full candidate pool. The result demonstrates that first-stage recall and shallow ranking quality should not be conflated.

Within Qwen3-Embedding, scaling is monotonic on Combined Recall@1000: 57.89% for the 0.6B model, 62.01% for the 4B model, and 64.53% for the 8B model. However, parameter scaling is not sufficient to determine cross-family performance: the 300M EmbeddingGemma model outperforms the 8B Qwen3 model on this metric. Late interaction is also inconsistent across families. mLateOn outperforms its dense counterpart mDenseOn, whereas pplx-embed-v1-late-0.6B trails pplx-embed-v1-0.6b. The advantage of late interaction is more visible at nDCG@10 than at Recall@1000, where both late-interaction models trail the strongest dense models at comparable scale.

Domain, language, and query-type effects

Aggregate scores conceal substantial heterogeneity. pplx-embed-v1-4b performs strongly across most domains, often matching or exceeding Nemotron-3-Embed-8B, while BM25 ranks last in every reported domain. Nemotron leads pplx-embed-v1-4b by two percentage points on Portuguese and one point on Russian, Italian, Spanish, and Korean, while matching or trailing it in the remaining languages. mDenseOn obtains 63% Recall@1000 on English but only 49–56% on other languages. mLateOn reaches 64% on English and 52–65% elsewhere, with Japanese slightly exceeding its English score.

The query-type split provides one of the clearest findings. Every neural retriever performs substantially better on primary queries than on supporting queries. The paper attributes this pattern partly to training-distribution mismatch: embedding models are commonly trained on human-written queries, which more closely resemble primary reformulations than the more exploratory, decompositional support queries issued by agents. This interpretation is plausible but not isolated experimentally from other factors, including support-query difficulty, query length, multilinguality, and the fact that support queries may target more specific or indirect evidence.

BM25 exhibits the opposite pattern, with supporting-query Recall@1000 approximately 1.9 percentage points higher than primary-query recall. The difference is small and does not overcome BM25’s low overall effectiveness, but it indicates that lexical matching may retain complementary value for machine-generated support queries whose salient entities or terms are explicit in the query.

Unique coverage and error structure

The paper’s most consequential claim is that aggregate recall does not determine ensemble value. BM25, despite the lowest Recall@1000, contributes 14,621 unique positives—documents not recovered by any other evaluated retriever. This is nearly six times the next-largest contribution: 2,451 unique positives from EmbeddingGemma-300M. mLateOn contributes 2,327, while pplx-embed-v1-4b and Nemotron-3-Embed-8B contribute only 884 and 705, respectively.

This result contradicts a simple strategy of selecting only the highest-scoring retrievers. The models with the best aggregate recall are comparatively redundant, whereas lexical retrieval recovers a much larger distinct portion of the judged relevant set. The result does not establish that BM25 should always be included in an ensemble, because the paper does not report a complete production ensemble study under fixed candidate budgets. It does establish that aggregate Recall@1000 is insufficient for selecting complementary retrievers.

The hard-negative analysis further characterizes the benchmark. Among documents ranked in the top 1,000 by all thirteen systems but labeled non-relevant, 15.8% are judged by the same LLM classifier to be relevant, indicating residual false negatives in the Combined set. Among the remaining hard negatives, the dominant categories are wrong aspect (39.0%), partial relevance (17.8%), related entity (13.8%), generic content (6.5%), and temporal mismatch (5.8%). These categories confirm that the corpus tests fine-grained constraint satisfaction rather than only broad semantic similarity.

The 6,424 hard positives—relevant documents missed by every evaluated retriever in the top 1,000—are dominated by lexical mismatch at 51.1%. Truncation accounts for 17.7%, multilingual mismatch for 15.4%, reasoning requirements for 7.0%, generic duplication for 4.1%, and entity or numeric constraints for 1.0%. The distribution implies that retrieval failures are not primarily caused by insufficient model scale. Lexical divergence, document truncation, and cross-lingual matching remain major failure modes even among large dense and late-interaction encoders.

Limitations and open questions

Q2D-Web’s principal limitation is restricted access. The corpus, queries, and judgments remain private to reduce training contamination, and only aggregate leaderboard results are exposed. This preserves benchmark validity more effectively than releasing all evaluation artifacts, but it limits independent auditing, reproducibility, error analysis, and examination of query-level variance. The benchmark is also derived from one commercial conversational search system, so its query distribution, citation policy, production ranking stack, and web corpus may not generalize to other agentic RAG deployments.

The relevance labels are not human gold judgments. Citation labels are sparse by construction, Web Ranking labels inherit system bias, and the Combined set relies partly on an LLM judge whose decisions may reflect stylistic preferences. The reported 15.8% false-negative rate among hard negatives directly demonstrates that the combined pool remains incomplete. In addition, the same LLM is used both to create part of the relevance set and to classify hard cases, so the error analysis is not independent validation.

The 512-token document truncation is another material constraint. Because the average document contains approximately 3,300 tokens, most document bodies are excluded from the main representation. The benchmark therefore measures retrieval under a fixed prefix-indexing protocol rather than unrestricted full-document retrieval. Finally, RRF subcorpus preservation is demonstrated for the tested models and Combined judgments, not guaranteed for future retrievers with different lexical, multilingual, or long-document behavior. The open empirical question is whether the substantial unique coverage of BM25 translates into consistent gains for fixed-budget hybrid ensembles, especially when candidate deduplication and downstream reranking costs are included.

Conclusion

Q2D-Web establishes a benchmark regime centered on the actual first-stage retrieval problem in agentic RAG: large heterogeneous corpora, machine-generated reformulations, multilingual queries, deep candidate recall, and multiple relevance signals. Its scale—approximately 190 million documents and 70,000 production-derived queries—addresses a gap left by benchmarks that provide either large corpora with few queries or many queries over relatively small collections.

The experiments show that model rankings are broadly stable across relevance sources, that RRF-based sampling preserves full-corpus ordering at roughly one-third of the corpus, and that high aggregate recall does not imply complementary retrieval coverage. The strongest operational conclusion is therefore not that one retriever dominates, but that evaluation must distinguish total recall, shallow ranking, query-type robustness, and unique positive recovery. Q2D-Web provides a framework for making those distinctions under production-scale constraints, while its private evaluation protocol leaves reproducibility and cross-system generalization as explicit unresolved questions.