How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation

Abstract: Evaluating retrieval-augmented generation (RAG) systems requires benchmarks that capture diverse question characteristics, yet practitioners lack empirical guidance on which dimensions to vary and at what granularity. We present HieraRAG, a hierarchical framework for studying granularity in RAG benchmark construction, defining optimal granularity as the level that maximizes discriminative power (the standard deviation of generation quality across categories) within a given RAG configuration. As a case study, we generate 5,872 synthetic question-answer (QA) pairs from FineWeb-10BT across 3 dimensions (Question Complexity, Answer Type, Linguistic Variation) at 3 granularity levels (2, 4, and 8 categories). With a BM25+Falcon-3-10B pipeline, optimal granularity varies by dimension: complexity benefits from fine-grained distinctions (discriminative power: 0.053) while answer type and linguistic variation peak at medium granularity. We introduce a Coherence Ratio metric to quantify whether fine-grained splits cleanly subdivide parent categories, revealing structural differences across dimensions (Question Complexity: 0.40 vs. Answer Type: 1.44). Human evaluation of 110 stratified QA pairs confirms synthetic quality. While these specific findings reflect a single configuration, HieraRAG provides a portable procedure and validation metric for practitioners to determine evaluation granularity within their own RAG settings.

• The paper presents HieraRAG, a framework that systematically categorizes synthetic QA along dimensions of question complexity, answer type, and linguistic variation.
• It demonstrates that optimal granularity varies by dimension, with fine splits enhancing complexity evaluation while medium splits better capture answer type and linguistic variation.
• The study introduces the Coherence Ratio metric as a diagnostic tool to balance discriminative power and structural alignment in benchmark design.

Hierarchical Evaluation for Retrieval-Augmented Generation: Principles and Insights from HieraRAG

Introduction

The paper "How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation" (2606.12789) presents HieraRAG, a principled framework for determining the optimal granularity of evaluation categories in RAG benchmark design. In the context of rapidly diversifying use cases for retrieval-augmented generation, there is a critical need for benchmarks that variably and systematically capture distinct QA properties—yet there is no consensus in the literature regarding which dimensions to diversify, or to what level of detail. HieraRAG operationalizes this variability by constructing a hierarchy over multiple question dimensions and providing empirical and metric-based tools for practitioners to choose granularity that maximizes discriminative power and interpretability.

Hierarchical Framework for Question Categorization

The authors define three foundational dimensions for question variation:

• Question Complexity (QC): Ranging from extractive and factoid questions to comparative synthesis and multi-hop reasoning.
• Answer Type (AT): Encompassing categories such as extractive, abstractive, summary, and explanation-based response types.
• Linguistic Variation (LV): Encoding degree of lexical and semantic shift between question and source material, from verbatim phrasing to high-level abstraction.

Each dimension is operationalized at three granularity levels: coarse (2 categories), medium (4), and fine (8), constructing a taxonomic tree for systematic analysis. Figure 1 provides a schematic for this hierarchy, illustrating the parent-child relationships and splitting across granularity.

Figure 1: Hierarchical structure of three question dimensions (QC, AT, LV) across three granularity levels. Each dimension subdivides from coarse (2 categories) to medium (4) to fine (8). User Expertise set to "Novice" for example QAs shown.

FineWeb-10BT [penedo2024fineweb] is leveraged to generate 5,872 synthetic question–answer pairs using DataMorgana (Filice et al., 22 Jan 2025), with the intended evaluation of each dimension decoupled by generating category-controlled batches per dimension. Assignment is intentionally unidimensional for experimental isolation.

Evaluation Metrics and the Coherence Ratio

To robustly assess RAG system performance and the utility of category granularity, the framework deploys standard IR and NLG metrics: MAP, nDCG@10, Recall@10 for retrieval, and Cosine Similarity (CS), ROUGE-1, BLEU for generation. Notably, it introduces Discriminative Power (standard deviation of CS across categories at a given granularity) and Normalized Mutual Information (NMI) as global measures.

A central methodological contribution is the Coherence Ratio ($\rho$): for each split from parent to child nodes within the hierarchy, $\rho$ quantifies the alignment and separation of sibling categories. This ratio is defined as the standard deviation of CS across siblings (horizontal signal) divided by deviation of their mean from the parent (vertical consistency), regularized by an $\epsilon$ factor. High $\rho$ indicates children are both distinct and representative of their parent category; low values suggest noisy, structurally incoherent splits.

Figure 2: Demonstrated Coherence Ratio ($\rho$) calculation for two medium to fine-grained splits within AT and LV dimensions. $\rho > 2.0$ indicates discriminative-yet-aligned children (preferred); $\rho < 1.0$ suggests poor hierarchical structure.

Core Empirical Findings

Dimension Discrimination

When comparing four dimensions at the coarse level, Linguistic Variation had the highest range in generation quality (CS), revealing it is the most discriminative under the tested BM25+Falcon-3-10B configuration. Notably, LV yielded the lowest absolute retrieval and generation metrics, emphasizing that discriminative power and absolute performance can diverge.

Effects of Granularity

Varying the number of categories per dimension, the discriminative power of category splits was not uniformly monotonic:

• QC: Discriminative power increased strictly from coarse to fine (0.007 → 0.053), indicating that finer complexity distinctions (such as between specific multi-hop or comparative reasoning types) reveal performance gaps otherwise masked at coarser levels.
• AT and LV: Discriminative power peaked at medium granularity (0.044 and 0.047, respectively) and decreased with further splitting, signifying that the additional distinctions introduce noise or redundancy rather than actionable insights.

The normalized MI across splits mirrored these trends. Crucially, in QC, the highest discriminative power at fine granularity was accompanied by low Coherence Ratio ($\rho$) (mean 0.40), signaling that these fine categories are not well-aligned structurally—useful diagnostically, but less principled as hierarchical taxonomy. AT, in contrast, had fine splits with high coherence (notably for summary_or_explanation), supporting refinement where justified by empirical alignment.

Interaction Effects and Human Validity

A 2×2 factorial between QC and LV showed additive, not interactive, effects: complexity mattered for performance only when vocabulary matched (i.e., retrieval succeeded), but in the presence of strong vocabulary shift, retrieval failure dominated and complexity distinctions were moot. Human validation confirmed high synthetic QA quality but highlighted that fine-grained categories are semantically ambiguous (29% inter-annotator agreement), underscoring the challenges of manual fine-labeling in open-domain settings.

Theoretical and Practical Implications

• Granularity as a Diagnostic Lever: The optimal granularity for benchmark construction is dimension-dependent. For complexity-oriented evaluation, fine-grained distinctions are diagnostic for current RAG models, justifying their use where nuanced reasoning failures are relevant. For answer type and linguistic variation, medium granularity is preferable; further splitting dilutes the performance signal.
• Structural Validation of Taxonomies: Introduction of the Coherence Ratio offers a scalable, automatic criterion for practitioners to prune and refine synthetic benchmark taxonomies. Low $\rho$ signals either semantically incoherent categories or over-fragmentation, and can preempt wasteful annotation efforts.
• Configurational Generality and Limitations: The empirical outcomes—i.e., which granularity maximizes discriminative power—are sensitive to system configuration and corpus characteristics (e.g., BM25's lexical bias amplifies LV's role). The methodological apparatus of HieraRAG, not specific splits or categories, is the principal contribution.
• Implications for Future RAG Development: As hybrid/dense retrievers or more robust generators are deployed, the axes and shape of optimal granularity are expected to shift. Integrating multi-label assignments and expanding to human-authored queries will provide greater ecological validity.

Conclusion

HieraRAG establishes a systematic, empirically-driven methodology for benchmark designers to determine and validate the granularity of evaluation axes in retrieval-augmented generation. By coupling hierarchical synthetic question generation with the Coherence Ratio and discriminative metrics, the framework surfaces both actionable diagnostic patterns and taxonomic ambiguities, equipping practitioners to build more interpretable and informative RAG benchmarks. The findings and tools provided in this work are broadly applicable and offer a template for extensible, configuration-specific benchmarking strategies in RAG system development.