HieraRAG: Hierarchical QA Benchmark Design

• HieraRAG is a hierarchical framework for designing synthetic QA benchmarks by partitioning question complexity, answer type, and linguistic variation into nested categories.
• It quantifies diagnostic power by measuring metrics such as standard deviation of generation scores and normalized mutual information to identify optimal granular splits.
• Empirical findings reveal that question complexity benefits from fine granularity, while answer type and linguistic variation perform best at medium granularity.

HieraRAG is a hierarchical framework for constructing and analyzing synthetic question-answering benchmarks for retrieval-augmented generation. Its central problem is not retrieval architecture design but benchmark design: which dimensions of question variation should be instantiated, and at what granularity, if the goal is to expose meaningful performance differences in a given RAG pipeline. It defines optimal granularity as the level that maximizes discriminative power, measured as the standard deviation of generation quality across categories, within a fixed configuration. In its case study, HieraRAG generates 5,872 synthetic QA pairs from FineWeb-10BT across three dimensions—Question Complexity, Answer Type, and Linguistic Variation—each instantiated at three nested levels of granularity: 2, 4, and 8 categories (Fensore et al., 11 Jun 2026).

1. Conceptual scope and problem definition

HieraRAG addresses a recurrent weakness in RAG evaluation: aggregate benchmark scores often conceal large performance differences across question types. The framework argues that benchmark construction has two coupled design variables. The first is the choice of dimensions of variation, such as complexity, answer format, or lexical mismatch. The second is the granularity at which those dimensions are partitioned. If categories are too coarse, diagnostically important failure modes are hidden; if they are too fine, categories may become redundant, noisy, or semantically unstable (Fensore et al., 11 Jun 2026).

The framework therefore treats granularity as an empirical variable rather than a taxonomic assumption. In the case study, each of the three core dimensions—Question Complexity (QC), Answer Type (AT), and Linguistic Variation (LV)—is defined hierarchically at coarse, medium, and fine resolutions, corresponding to 2, 4, and 8 categories. The hierarchy is intended to be nested: each fine category specializes one medium parent, and each medium category subdivides one coarse parent. Categories within a level are mutually exclusive for generation, although the subsequent coherence analysis shows that semantic overlap can still emerge in practice (Fensore et al., 11 Jun 2026).

A common misconception is to read HieraRAG as a retrieval model. The framework is instead a benchmark-construction and benchmark-analysis methodology. Its hierarchy is a hierarchy of evaluation categories, not a hierarchy of indexed documents, graph nodes, or retrieval stages. This distinction is essential for situating the method relative to the broader hierarchical RAG literature.

2. Hierarchical benchmark construction

The source corpus in the case study is FineWeb-10BT, described as a web corpus with 10B tokens and approximately 15M documents. Documents are randomly sampled from this corpus for synthetic QA generation. For retrieval in the evaluation pipeline, the same corpus is chunked into 512-token chunks, indexed with PyTerrier, and queried with BM25 at $k=10$ (Fensore et al., 11 Jun 2026).

Synthetic generation uses DataMorgana, which the paper describes as a tool for creating diverse synthetic QA benchmarks by leveraging Claude 3.5 Sonnet. For each generated instance, the system produces a synthetic question, a reference answer, and a source document ID. That document identifier is critical because it supports both retrieval evaluation and answer-quality evaluation. HieraRAG is explicitly presented as portable: the framework is not tied to DataMorgana, even though DataMorgana is the concrete generator used in the case study (Fensore et al., 11 Jun 2026).

The benchmark construction is organized around three research questions. For RQ1, the framework generates 1,600 questions, with 400 per dimension, to compare which dimensions are most discriminative at coarse granularity; the dimensions in that stage are QC, AT, LV, and Question Phrasing. For RQ2, it generates 3,272 questions across QC, AT, and LV at all three granularity levels. For RQ3, it generates 1,000 questions in a $2\times 2$ factorial design crossing LV and QC, with about 250 questions per cell (Fensore et al., 11 Jun 2026).

A methodological constraint is that, for RQ1 and RQ2, each synthetic question is assigned to one dimension only during generation. The paper states that this was done to isolate the effect of one dimension at a time, while also acknowledging that real questions vary along multiple dimensions simultaneously. The stated implication is that uncontrolled variation in the other dimensions remains a confounder; future work is suggested in the form of full factorial coverage or post-hoc multi-label tagging (Fensore et al., 11 Jun 2026).

User expertise is controlled rather than made a primary axis of the hierarchy. The generation process sets expertise to 50% novice and 50% expert. This is intended to prevent expertise effects from dominating the measured impact of question characteristics (Fensore et al., 11 Jun 2026).

3. Granularity hierarchy and validation metrics

The central analytic notion in HieraRAG is discriminative power. For each granularity level within a dimension, the framework computes the standard deviation of category-level cosine-similarity generation scores. In the paper’s notation, this is reported as DiscPow = $\sigma(\mathrm{CS})$ across categories. Higher values indicate that the partition is surfacing stronger performance differentials and is therefore more diagnostically useful (Fensore et al., 11 Jun 2026).

Discriminative power is supplemented by Normalized Mutual Information (NMI), used to measure association between category assignments and performance bins, and by the Coherence Ratio, which is intended to test whether fine-grained splits are structurally valid refinements of their parents. The Coherence Ratio is built from two components. The first, $\sigma_{\text{sib}}$, is the standard deviation of cosine similarity across sibling categories and represents horizontal discrimination. The second, $\delta_{\text{vert}}$, is the absolute difference between the parent score and the mean child score and represents vertical consistency. The paper also specifies $\epsilon = 0.001$ to prevent division by zero, and gives heuristic thresholds in which high $\rho$ with $\rho > 2.0$ is preferred and low $\rho$ with $\rho < 1.0$ indicates poor hierarchical structure (Fensore et al., 11 Jun 2026).

The hierarchy is analyzed through parent–child splits. Each dimension has 6 splits total: 2 coarse-to-medium splits and 4 medium-to-fine splits. A split is not evaluated merely on whether its children differ, but on whether the difference is achieved without drifting too far from the parent’s performance profile. This makes the Coherence Ratio a structural diagnostic rather than a pure difficulty-separation metric (Fensore et al., 11 Jun 2026).

The evaluation pipeline for the case study is deliberately simple. Retrieval is BM25 over 512-token chunks indexed with PyTerrier; generation uses Falcon-3-10B-Instruct at temperature 0.6, with instructions to refuse when information is insufficient. Retrieval metrics include MAP, nDCG@10, and Recall@10; generation metrics include cosine similarity using MiniLM-L6-v2 embeddings, ROUGE-1, and BLEU. The main reported generation analyses use cosine similarity (Fensore et al., 11 Jun 2026).

4. Empirical findings on optimal granularity

The core empirical result is that optimal granularity is dimension-dependent. Question Complexity benefits from increasingly fine distinctions, while Answer Type and Linguistic Variation peak at medium granularity. This is the paper’s principal substantive claim and the main reason it argues against a universal benchmark taxonomy (Fensore et al., 11 Jun 2026).

Dimension	Best granularity	Key evidence
Question Complexity	Fine (8 categories)	DiscPow rises from 0.007 to 0.035 to 0.053
Answer Type	Medium (4 categories)	DiscPow rises from 0.024 to 0.044, then drops to 0.037
Linguistic Variation	Medium (4 categories)	DiscPow rises from 0.039 to 0.047, then drops to 0.031

For QC, the gains are monotonic. Discriminative power moves from 0.007 at coarse granularity to 0.035 at medium and 0.053 at fine. NMI also increases monotonically from .008 to .027 to .040. The paper interprets this as evidence that a binary simple/complex split underrepresents meaningful variation in reasoning demand. Fine-grained distinctions expose performance differences that the coarse hierarchy suppresses (Fensore et al., 11 Jun 2026).

For AT, medium granularity is optimal. Discriminative power goes from 0.024 to 0.044 and then declines to 0.037, a 16% drop from medium to fine. The paper’s explanation is that some fine splits are coherent, but not enough of them are consistently strong to justify the 8-way taxonomy. One named example is the summary_or_explanation split, which has coherence 3.31, while the other medium-to-fine AT splits average only 0.82 (Fensore et al., 11 Jun 2026).

For LV, medium granularity is also optimal. Discriminative power is 0.039 at coarse, 0.047 at medium, and 0.031 at fine, a 33% decrease from medium to fine. NMI likewise drops sharply at the fine level, from .054 at medium to .020 at fine. The paper attributes this to unstable or semantically drifting subcategories at the finest resolution. It highlights conceptual_rephrase as the poorest calibrated split, with $2\times 2$0 (Fensore et al., 11 Jun 2026).

The coherence analysis adds a second layer to these findings. QC has mean coherence 0.40, whereas AT has mean coherence 1.44. The paper presents this contrast as structurally important: QC is the most diagnostically useful dimension at fine granularity, but it is not the cleanest hierarchy. AT is less diagnostically extreme, yet its hierarchy behaves more like a well-formed taxonomy. A plausible implication is that diagnostic value and hierarchical well-formedness are partially independent design objectives (Fensore et al., 11 Jun 2026).

At coarse granularity in RQ1, LV is the most discriminative dimension, with a cosine-similarity range of 0.077, while QC appears weak, with a range of 0.010. The later RQ2 result reverses the interpretive status of QC: complexity becomes the strongest beneficiary of fine granularity once its internal structure is expanded. This is one of the clearest demonstrations that coarse benchmark partitions can misrepresent the diagnostic value of a dimension (Fensore et al., 11 Jun 2026).

Human validation covers 110 stratified QA pairs and is reported as confirming synthetic quality. The paper presents this as a quality-control check on the synthetic generation process rather than as the main empirical result (Fensore et al., 11 Jun 2026).

5. Position within hierarchical RAG research

HieraRAG belongs to the hierarchical RAG literature only in a qualified sense. Its contribution lies in hierarchical benchmark design, not in hierarchical retrieval or generation architecture. That distinguishes it from retrieval-time systems such as HiRAG, which constructs a multi-layer hierarchical knowledge graph and uses local, global, and bridge retrieval over that graph (Huang et al., 13 Mar 2025). It also differs from KohakuRAG, which preserves document structure through a four-level tree representation of document $2\times 2$1 section $2\times 2$2 paragraph $2\times 2$3 sentence with bottom-up embedding aggregation (Yeh et al., 8 Mar 2026).

The contrast is equally sharp with parent–child retrieval systems. H-RAG separates fine-grained child-level retrieval from parent-level context reconstruction during generation, using overlapping sentence-based child chunks and full documents as parents (Elchafei et al., 1 May 2026). A-RAG, by contrast, frames hierarchy as a tool interface over keyword search, semantic search, and chunk read, enabling an agent to move across multiple granularities during retrieval (Du et al., 3 Feb 2026). HF-RAG uses yet another notion of hierarchy: layered fusion of multiple rankers within each source, followed by cross-source normalization and fusion (Santra et al., 2 Sep 2025).

There is also a generator-side interpretation of hierarchy. HIRAG defines three progressively hierarchical abilities for RAG generation—Filtering, Combination, and RAG-specific reasoning—and instruction-tunes the generator to execute them via a multi-level “think before answering” process (Jiao et al., 8 Jul 2025). HieraRAG does not operate at this level either. Its hierarchy is neither a document tree, nor a graph abstraction, nor a retrieval controller, nor a staged generator policy.

The result is that the term “hierarchical RAG” spans several non-equivalent design axes in the current literature: hierarchical document indexing, hierarchical graph abstraction, parent–child retrieval, layered evidence fusion, agentic multi-granularity search, generator-side capability decomposition, and benchmark taxonomy design. HieraRAG is the clearest representative of the last category.

6. Limitations, interpretation, and implications

The framework is explicit that its findings are configuration-specific. The headline conclusion—that QC prefers fine granularity while AT and LV prefer medium granularity—comes from a BM25 + Falcon-3-10B pipeline over FineWeb-10BT. The paper describes the procedure as portable, but does not claim that the resulting optimal granularities transfer unchanged to other retrievers, generators, corpora, or application domains (Fensore et al., 11 Jun 2026).

Another limitation is the single-dimension assignment used for most of the synthetic generation. Because each question is controlled along one target dimension at a time, uncontrolled variation in the other dimensions remains. The paper states this directly and suggests fuller factorial designs or post-hoc multi-label classification as future extensions (Fensore et al., 11 Jun 2026).

The coherence results also complicate a naïve reading of finer granularity as inherently better. QC attains the highest discriminative power at fine granularity, yet its mean coherence is only 0.40. AT, by contrast, is more structurally coherent at 1.44, even though its best granularity is medium rather than fine. This suggests that benchmark designers may need to trade off diagnostic sharpness against taxonomic cleanliness, rather than maximizing only one criterion (Fensore et al., 11 Jun 2026).

A further implication concerns evaluation methodology in RAG research more broadly. Many hierarchical RAG systems claim gains on complex reasoning, lexical mismatch, or answer-format sensitivity, but benchmark category design is often treated as fixed. HieraRAG suggests that some reported strengths or weaknesses may partly depend on whether the benchmark partitions those phenomena at a granularity capable of revealing them. In that sense, the framework functions as a meta-evaluation method for RAG systems rather than as a competing retriever (Fensore et al., 11 Jun 2026).

Taken as a whole, HieraRAG establishes that benchmark granularity is itself a measurable property of RAG evaluation. Its main contribution is to convert a largely intuitive benchmarking choice into a hierarchical, empirically testable design problem.