PRGB Benchmark: Fine-Grained RAG Evaluation

• The PRGB Benchmark is a multi-level evaluation framework that decouples LLM internal knowledge from retrieved documents using dynamic placeholder substitutions.
• It measures LLM capabilities in noise filtering, document integration, and multi-hop reasoning, providing granular insights under varied retrieval conditions.
• Experimental analysis shows that while larger models excel at synthesis, they are challenged by increased retrieval noise, highlighting the need for robust RAG designs.

The PRGB Benchmark is a multi-level, fine-grained evaluation framework tailored to assessing Retrieval-Augmented Generation (RAG) systems, with a particular focus on decoupling LLM parametric knowledge from the utilization of retrieved external documents. Unlike prior benchmarks emphasizing global or retrieval-side performance, PRGB systematically targets the intrinsic capabilities of LLMs in synthesizing, filtering, and reasoning over external context. A central methodological innovation is its placeholder-based approach, which dynamically substitutes factual values in reference documents, compelling models to rely on retrieval rather than internal memorization. The benchmark provides public code and a reproducible pipeline for granular evaluation of document-level integration, reasoning, and robustness in diverse RAG scenarios (Tan et al., 23 Jul 2025).

1. Motivation and Conceptual Framework

PRGB Benchmark was motivated by the lack of systematic, granular RAG evaluation methods that isolate an LLM’s ability to leverage retrieval context independently of its pre-trained parametric content. Most existing RAG benchmarks assess system-wide outcomes, such as answer accuracy or broad robustness to noise, without clarifying whether correct answers stem from effective external document use or from model memorization. This ambiguity poses challenges for understanding error propagation, context faithfulness, and the boundary between retrieval and generation functions.

The key conceptual advance in PRGB is the dynamic placeholder substitution mechanism. In this framework, variable factual values within golden reference documents are replaced with explicit placeholder tokens. During evaluation, these placeholders are resolved with candidate values derived from the question and retrieval context. This design forces the LLM to extract, combine, and reason from retrieved documents rather than defaulting to internal knowledge, robustly measuring document dependence.

2. Progressive Dimensions of Evaluation

PRGB’s evaluation protocol is organized along three progressive dimensions, each probing a distinct aspect of LLM document-grounded reasoning:

1. Multi-Level Filtering Abilities: The LLM is presented with a set of retrieved documents, only some of which are relevant (“golden”). The model must discriminate against various forms and degrees of noise:
   • Weak noise: Irrelevant documents.
   • Moderate noise: Documents from similar entities.
   • Hard noise: Documents from the parent entity or broadly related contexts. Performance is assessed by the model’s accuracy in filtering correct context as noise levels increase.
2. Combination Abilities: This dimension examines whether the LLM can accurately synthesize or compose outputs from multiple disparate retrieved documents. Types include:
   • Explicit composition: Combining facts across documents (e.g., junction of two triplets).
   • Multi-value composition: Assembling multi-part answers from several sources.
   • Multi-scenario composition: Integrating context across hierarchical entity sets.
3. Reference Reasoning: Beyond extraction, this domain tests multi-hop inferential power and indirect deduction over the retrieved context:
   • Attribute comparison across entities.
   • Relationship-based value reasoning (e.g., deducing a value from document structure). The evaluation measures both correctness and faithfulness to the non-parametric knowledge provided.

Triplet-style metadata (parent entity $E^p$, child entities $e^p_{ij}$, and triplets $(e^p_{ij}, p, v)$) forms the backbone of both task instantiation and document synthesis, with task-specific variations mapped onto these core structures.

3. Placeholder-Based Decoupling Strategy

PRGB’s distinguishing methodological contribution centers on its placeholder-based document construction. For each golden document, factual values are programmatically replaced by one or more placeholder tokens, resulting in template triplets such as $(e, p, \text{Placeholder})$.

Synthetic documents populated with placeholders are generated using advanced autoregressive methods (e.g., high-capacity models like GPT-4o and Qwen2.5-MAX) to preserve fluency and compositional diversity. Noisy documents are constructed in parallel, following the same triplet and substitution logic, but drawing from distractor triplets or entity variants as described in the benchmark protocol.

At inference time, when evaluating LLM responses, placeholders in the generated output are dynamically resolved via context-grounded candidate extraction. By controlling for parametric content and shifting the answer space to contextual values, this approach robustly measures the degree of document reliance, mitigates internal knowledge contamination, and supports repeated, statistically sound evaluation.

4. Experimental Analysis and Model Limitations

Empirical studies utilizing the PRGB Benchmark have revealed notable limitations in contemporary LLMs embedded in RAG architectures:

• Error Resilience: LLMs often degrade in context filtering under increased noise, especially when progressing from weak to hard distractor documents. This manifests as significant accuracy drops in noisy retrieval settings, diagnostic of brittle document relevance modeling.
• Context Faithfulness: Larger models, despite improved compositional reasoning, may default to paraphrase heuristics or omit critical context details (e.g., truncating dates or objects), undermining fidelity to the retrieved evidence.
• Needle-in-the-haystack vs. Synthesis: Smaller models can outperform larger ones on simple extraction tasks but struggle on complex multi-document synthesis; conversely, larger LLMs excel at integrating disparate sources in complex reasoning tasks, albeit sometimes at the expense of exactness.

These findings demonstrate the necessity for granular, document-focused benchmarks beyond naïve QA metrics or end-to-end scorecards, confirming that model–retrieval decoupling is vital to advance reliability.

5. Reproducibility, Pipeline, and Resources

PRGB prioritizes reproducibility in both task setup and result measurement. The full benchmark suite, including task configuration, placeholder substitution procedures, noise corpus generation, and model interface pipelines, is publicly released for community use:

• GitHub repository: https://github.com/Alipay-Med/PRGB

Hyperparameter schemas, document construction methods, and evaluation metrics are documented in detail for direct adoption or extension. Benchmarked models can be tested in a standardized workflow, enabling cross-institutional, repeated benchmarking with controlled context exposure.

6. Implications and Prospects for RAG System Development

By promoting fine-grained, decoupled evaluation of document grounding in RAG systems, PRGB sets a precedent for rigorous benchmarking methodologies in production and academic environments. Immediate implications include:

• Design of RAG systems with explicit context integration and minimal hallucination.
• Development of LLM architectures or retrieval strategies tuned for robust filtering, compositional reasoning, and faithfulness rather than mere parametric recall.
• Comparative studies on retrieval strategies, document noise robustness, and placeholder-based answer fidelity.

A plausible implication is that future RAG system improvements will concentrate on document context exploitation, fine-tuned noise filtering, and reasoning mechanisms validated through tasks and metrics like those provided by PRGB.

7. Summary

PRGB Benchmark represents a significant methodological advance in RAG evaluation, offering a reproducible, multi-dimensional platform for granular analysis of document integration, filtering, and reasoning in LLMs. Through its placeholder-based approach and progressive task dimensions, PRGB provides objective, fine-grained insight into LLM performance under varied retrieval conditions, laying the groundwork for the next generation of reliable, externally grounded RAG systems (Tan et al., 23 Jul 2025).