PRGB: Fine-Grained RAG Benchmark

• PRGB is a benchmark that isolates the LLM’s ability to use retrieved documents by employing a dynamic placeholder substitution mechanism.
• It evaluates RAG systems across three dimensions—multi-level filtering, combination, and reference reasoning—to test evidence binding and noise rejection.
• The bilingual benchmark integrates synthetic and human-verified metadata from Wikipedia, ensuring robust evaluation under controlled dynamic instantiation.

Placeholder-RAG-Benchmark, introduced as PRGB, is a multi-level, fine-grained benchmark for Retrieval-Augmented Generation that is designed to evaluate how well the LLM component of a RAG system actually uses retrieved documents rather than relying on parametric recall. It is built on triplet-based meta-data $(e,p,v)$, three progressive capability dimensions—multi-level filtering, combination, and reference reasoning—and a dynamic placeholder substitution mechanism intended to decouple parametric knowledge from external knowledge. The benchmark is bilingual, covering English and Chinese, and is explicitly structured to probe document utilization, error resilience, and context faithfulness under controlled noise and reasoning conditions (Tan et al., 23 Jul 2025).

1. Definition and evaluative objective

PRGB was introduced to address a gap in RAG evaluation: most benchmarks measure overall pipeline performance, but they do not isolate the generator’s ability to use retrieved documents. In a standard RAG setting, a retriever returns a subset $R(q) \subset \mathcal{D}$ for query $q$, and the generator models $p(y \mid q, R(q))$. A high final QA score can therefore conflate strong retrieval, memorized parametric knowledge, and genuine grounding. PRGB targets that ambiguity by focusing on the LLM’s document utilization under controlled conditions (Tan et al., 23 Jul 2025).

The benchmark is organized around three progressive dimensions. The first is multi-level filtering, which tests whether the model can select the correct information when golden documents are mixed with weak, moderate, and hard noise. The second is combination, which tests whether the model can aggregate information across entities, attributes, or scenarios. The third is reference reasoning, which covers multi-hop, indirect, comparative, and deductive reasoning where the answer is not directly copied from a document. This tiered design makes PRGB less a single dataset than a structured evaluation suite for distinct subskills inside RAG generation (Tan et al., 23 Jul 2025).

A central design goal is to assess the LLM’s role inside RAG rather than the end-to-end retriever. The benchmark assumes that retrieval has already produced the candidate context, though its document library can also be repurposed for retriever testing. This suggests that PRGB is best understood as a generator-centric complement to retriever benchmarks and deployment-level RAG harnesses rather than a replacement for them.

2. Placeholder-based methodology

The core mechanism is a dynamic placeholder framework built on triplets $(e,p,v)$, where $e$ is an entity, $p$ a predicate, and $v$ a value. For a golden triplet, the benchmark replaces the value with a symbolic placeholder, yielding a placeholder-based triplet such as

$(e,p,\text{Placeholder}).$

If the value is multi-valued, multiple placeholders are used. The resulting synthetic document contains the logical structure of the fact while withholding the surface realization of the answer value (Tan et al., 23 Jul 2025).

PRGB then generates a candidate value set

$P_N = \{P_1, P_2, \dots, P_N\},$

with datatype-matched candidates, exactly one of which corresponds to the true value or true subset. The benchmark repeatedly instantiates placeholders with different candidate assignments across runs. Because the document text remains nearly identical while the placeholder mapping changes, a model that answers correctly must track the mapping induced by the retrieved context rather than emit a memorized fact. This is the benchmark’s principal device for decoupling parametric knowledge from retrieved knowledge (Tan et al., 23 Jul 2025).

This mechanism is not simple cloze masking. The documents are synthesized around entities using GPT-4o and Qwen2.5-MAX, with golden documents containing placeholders and noise documents retaining incorrect values. Dynamic substitution is then used to test whether the answer changes appropriately when the candidate mapping changes. A plausible implication is that PRGB functions as a symbolic binding test inside a RAG setting: it evaluates whether the model binds retrieved evidence to answer variables consistently across perturbations.

3. Capability dimensions and task taxonomy

PRGB defines three major dimensions, each with internal subtypes that progressively increase document-selection and reasoning demands (Tan et al., 23 Jul 2025).

Dimension	Task types	Stress target
Multi-level filtering	weak, moderate, hard noise	noise rejection and document selection
Combination	explicit, multi-value, multi-scenario	evidence aggregation across entities or attributes
Reference reasoning	comparative, deductive, comparative deductive	indirect inference and multi-hop grounding

In multi-level filtering, the answer is directly contained in a golden document, but the model must identify it under different noise regimes. Weak noise uses triplets from irrelevant parent entities. Moderate noise uses semantically close entities from the same parent and propagation dimension. Hard noise uses parent-level triplets that encode general cases conflicting with child-level exceptions. The benchmark’s Mozart example illustrates the failure mode: the golden document gives November 9, 1786, while a noisy document gives October 7, 1783, and a strong model can still answer with the noisy date (Tan et al., 23 Jul 2025).

In combination, the benchmark tests whether the model can integrate information across multiple documents. Explicit composition asks for ordered multi-entity outputs such as host countries of the 2016 and 2020 Olympics. Multi-value composition partitions a set-valued predicate, such as host cities, across multiple documents and requires union over values. Multi-scenario composition asks about a parent entity whose answer must be assembled from multiple child entities that share the same predicate. These tasks are intended to separate simple retrieval from actual evidence aggregation (Tan et al., 23 Jul 2025).

In reference reasoning, answers are not directly present as span copies. Comparative reasoning requires reading values and comparing them. Deductive reasoning-1 uses inheritance, as in inferring Harvard’s borrowing fee from the rule for Ivy League schools. Deductive reasoning-2 uses relationship-based rules, as in inferring that the Guangzhou-to-Shenzhen expressway is free because both cities are in the same province and the rule states that expressways within the same province are free. Comparative deductive reasoning combines abstraction and comparison, such as recognizing “airbags” and “automatic alarms” as shared safety measures across entities (Tan et al., 23 Jul 2025).

4. Dataset construction and benchmark composition

The benchmark is built from structured triplet metadata

$R(q) \subset \mathcal{D}$0

where $R(q) \subset \mathcal{D}$1 is a parent entity, $R(q) \subset \mathcal{D}$2 a propagation dimension, and $R(q) \subset \mathcal{D}$3 a child entity. After manual verification, the triplet store contains 224 parent entities, 2,272 child entities, and 16,033 triplets. These were drawn from Wikipedia categories such as sports events, awards, animals, and fictional entities, and then used to synthesize documents and tasks (Tan et al., 23 Jul 2025).

PRGB is bilingual. The English dataset contains 3,887 samples, and the Chinese dataset contains 3,387 samples. In the English dataset, the subtask breakdown is 816 filtering, 2,042 combination, and 1,029 reasoning samples. In the Chinese dataset, it is 800 filtering, 1,495 composition, and 1,092 reasoning samples. The average number of golden documents, weak noise documents, moderate noise documents, hard noise documents, and candidate placeholder values is also reported separately for English and Chinese, indicating that the two languages are not simple translations but separately instantiated benchmark suites (Tan et al., 23 Jul 2025).

The benchmark uses substantial human quality control despite its synthetic construction. Multiple state-of-the-art LLMs are run on the easiest setting, samples where multiple models fail are identified, and about 30% are manually reviewed and corrected to ensure that placeholder substitutions do not create contextual contradictions and that placeholder–answer relationships are correct. This curation step is important because the validity of the benchmark depends on the synthetic documents remaining logically consistent under dynamic substitution (Tan et al., 23 Jul 2025).

A common misconception is that PRGB is a purely synthetic benchmark with no validation. In fact, the synthetic richness is paired with manual verification, bilingual task construction, and explicit noise design. What remains synthetic is primarily the controlled document generation process, not the evaluation logic.

5. Evaluation protocol, metrics, and empirical findings

The evaluation algorithm runs each data point against multiple placeholder instantiations. For dataset $R(q) \subset \mathcal{D}$4 and placeholder candidate set $R(q) \subset \mathcal{D}$5, the benchmark performs repeated inference over $R(q) \subset \mathcal{D}$6 candidate instantiations and averages the metric values. The main experiments use $R(q) \subset \mathcal{D}$7 noise docs = 4, $R(q) \subset \mathcal{D}$8 noise docs = 4, $R(q) \subset \mathcal{D}$9 noise docs = 1, and number of placeholders per data point = 3 (Tan et al., 23 Jul 2025).

Two principal metric families are used. The first is Accuracy (Covered Exact Match), which checks whether required answer keywords appear in the output under logical OR and AND conditions; it supports synonyms, multi-item answers, and decomposed phrases. The second is GPT-based evaluation, in which GPT-4o judges whether the answer is correct given the question, documents, and model answer. Metrics are averaged over placeholder instantiations, so a model that only occasionally tracks the placeholder mapping is penalized (Tan et al., 23 Jul 2025).

The benchmark evaluates both proprietary and open-source models. Reported systems include Gemini-2.5-pro-preview, Gemini-2.5-flash-preview, Claude-3.7-sonnet, GPT-4.1, GPT-4o-1120, multiple Qwen2.5 and Qwen3 variants, DeepSeek-V3, Gemma3, and Hunyuan-80B-A13B. In the Chinese setting, the highest overall score is 87.33 for Gemini-2.5-pro-preview; in English it is 84.89 for the same model. Across many models, filtering scores can reach ~97–98%, while reasoning remains substantially lower, often in the 40–75% range, indicating that document selection is much easier than indirect grounded reasoning (Tan et al., 23 Jul 2025).

The results emphasize several recurring patterns. Reasoning-enabled models perform better than non-reasoning versions across filtering, combination, and reference reasoning. Filtering does not scale monotonically with model size, because smaller models sometimes copy exact phrases more reliably, whereas larger models paraphrase and occasionally omit crucial detail. By contrast, larger models are clearly better at combination and reasoning. In the English combination tasks, Qwen2.5-7B scores 74.07% on explicit composition, 39.66% on multi-value composition, and 28.60% on multi-scenario composition, while Qwen2.5-MAX reaches 86.53%, 73.06%, and 75.20% respectively. In reasoning, Qwen2.5-7B reaches 26.92% overall, compared with 55.37% for Qwen2.5-MAX (Tan et al., 23 Jul 2025).

Placeholder perturbation experiments add a further diagnostic. With three placeholders per sample, the proportion of completely correct answers increases with model size, fully incorrect cases decrease, and partially correct outcomes remain around ~30% and grow somewhat with size. This suggests that larger models are more stable under dynamic substitution but still do not consistently track every placeholder change (Tan et al., 23 Jul 2025).

6. Position in the RAG benchmark landscape, limitations, and significance

PRGB occupies a specific position within the broader RAG evaluation ecosystem. Unlike SCARF, which is a black-box benchmarking harness for deployed RAG systems accessed through REST APIs and oriented toward framework, vector database, and serving-stack comparisons (Rengo et al., 10 Apr 2025), PRGB isolates the LLM’s document-utilization behavior under controlled contexts (Tan et al., 23 Jul 2025). Unlike RAGBench, which formalizes the TRACe framework around relevance, utilization, adherence, and completeness across 100k examples (Friel et al., 2024), PRGB uses placeholders to test whether the generator truly binds retrieved evidence to answer content. GaRAGe emphasizes long-form answers, relevance-aware factuality, attribution, and deflection over annotated web and private grounding (Sorodoc et al., 9 Jun 2025), while MIGRASCOPE is retriever-centric and information-theoretic, measuring divergence, redundancy, synergy, and marginal contribution among retrievers (Zheng et al., 25 Feb 2026). In that landscape, PRGB is best understood as a generator-centric, fine-grained diagnostic benchmark rather than an end-to-end deployment harness, a retriever benchmark, or a long-form grounding benchmark.

The benchmark also differs from domain-specific retrieval benchmarks such as TeleEmbedBench, which focuses on telecom embedding models, Top-1 Retrieval Accuracy, MRR@5, and robustness to noisy user queries (Gajjar et al., 20 Apr 2026), and from automated benchmark-generation frameworks such as IRB, which constructs fresh factual QA datasets from citation-backed Wikipedia scaffolds (Do et al., 8 Feb 2026). PRGB does not seek to automate benchmark refresh or specialize to one corpus; its novelty lies in the placeholder mechanism and the decomposition of RAG generation into filtering, combination, and reasoning tasks.

Its limitations are explicit. Current entities and triplets are drawn from popular Wikipedia categories, so domain coverage remains broad but not specialized. Only English and Chinese are supported. The tasks are primarily QA-like and do not directly cover long-form summarization or code-grounded reasoning. Placeholder construction and candidate alignment must be carefully maintained, and the evaluation metrics are primarily accuracy-based and GPT-judged correctness. Future work proposed in the benchmark includes more tasks and domains, more nuanced metrics for trustworthiness and explanation quality, additional languages, and better interfaces for plugging the benchmark into real-world RAG systems (Tan et al., 23 Jul 2025).

The main significance of Placeholder-RAG-Benchmark is therefore methodological. It shifts evaluation away from holistic RAG accuracy toward a controlled test of whether a model can filter noise, combine distributed evidence, and perform grounded reference reasoning while remaining sensitive to dynamic placeholder assignments. That focus makes it particularly useful wherever the central research question is not merely whether a RAG system answers correctly, but whether the generator is genuinely using the retrieved documents to do so (Tan et al., 23 Jul 2025).