---
title: 'RedQA: Redundancy-Aware QA Benchmark'
url: https://www.emergentmind.com/topics/redqa
type: topic
---

# RedQA: Redundancy-Aware QA Benchmark

RedQA is a redundancy-aware retrieval and question-answering benchmark produced by the RARE framework for high-similarity corpora such as financial filings, legal codes, and patents. It was created to evaluate retrieval-augmented generation under conditions where the same fact may appear in many near-duplicate passages, so a retriever may return a passage different from the annotated source chunk yet still provide sufficient evidence. Unlike standard benchmarks that assume distinct documents, low overlap, and one canonical supporting passage, RedQA labels semantically equivalent evidence at the atomic-fact level and evaluates sufficiency rather than exact passage match [2604.19047].

## 1. Motivation and benchmark philosophy

RedQA is motivated by a mismatch between standard QA evaluation and enterprise-style retrieval environments. The underlying claim is that most conventional QA and retrieval benchmarks assume distinct passages or documents, low overlap between evidence sources, and a single correct supporting passage. In high-similarity corpora, that assumption becomes brittle: the same fact may recur across multiple passages, retrievers may return a different but equally valid passage than the canonical annotation, and standard evaluation can therefore underestimate retrieval quality. The paper also argues the converse: systems that perform well on low-overlap benchmarks may generalize poorly to corpora with strong inter-document similarity and redundancy [2604.19047].

The benchmark therefore redefines what counts as relevant evidence. If a question is generated from a source chunk \(P_i\), the answer may also appear in other chunks \(P_j\), \(j \neq i\). Under RedQA, retrieving \(P_j\) is still correct if it supports the answer. This design shifts evaluation from exact passage identity to evidence sufficiency. In practical terms, the benchmark asks whether the retrieved set contains the information required to answer the question, not whether it reproduces a single annotated provenance path.

This design choice is especially consequential for multi-hop retrieval. In a redundant corpus, top-\(k\) retrieval can be dominated by passages that repeat the same fact, leaving complementary facts unretrieved. The paper describes this as an **overlap-induced evidence bottleneck**: redundancy may provide alternative evidence paths, but high similarity can also crowd out the diverse evidence required for compositional answering [2604.19047].

## 2. RARE framework and atomic-fact construction

RedQA is instantiated through RARE, short for **Redundancy-Aware Retrieval Evaluation**, which constructs the benchmark in three stages: **Valid Information Selection**, **Systematic Redundancy Tracking**, and **Question–Answer Generation** [2604.19047].

A central operation is document decomposition into atomic information units. Each input chunk is decomposed by an LLM into atomic facts that are defined as **single-claim**, **self-contained**, suitable for **fine-grained redundancy tracking**, and useful as building blocks for **multi-hop question generation**. These atoms are then filtered by a binary LLM judge \(\phi(\cdot)\in\{Pass,Fail\}\). A valid atom must be complete and self-contained, non-trivial and utility-relevant, and factual rather than metadata or document structure.

Valid atoms are ranked by five criteria: **Validity**, **Completeness**, **Specificity**, **Clarity**, and **Questionability**. To improve multi-criteria LLM judgment, the framework introduces **CRRF**, or **Criterion-wise Prompting with Reciprocal Rank Fusion**. CRRF evaluates each criterion independently and fuses the resulting ranks by

$$
s(x) = \sum_{i=1}^{N} \frac{1}{\mathrm{rank}_i(x)}.
$$

The stated rationale is that separate prompting reduces cross-criterion interference, while rank-based fusion is more robust than averaging raw confidence scores. In ablation, **Separate + RRF** is reported as the best-performing setup, achieving **NDCG@3 = 0.463** on GPT-5 Nano and **NDCG@3 = 0.467** on GPT-5, and it is described as more stable across runs than joint prompting [2604.19047].

Systematic redundancy tracking proceeds in two stages. For each target atomic fact \(a_t\), the framework first retrieves high-recall candidates by embedding similarity:

$$
\mathcal{C}_\tau(a_t) = \bigl\{\, a_j \in \mathcal{A} \ \bigm|\ \mathrm{chunk}(a_j)\neq \mathrm{chunk}(a_t),\ \text{sim}(a_t,a_j) \ge \tau \,\bigr\},
$$

with a recall-oriented threshold \(\tau = 0.5\). It then refines this set through an LLM verifier:

$$
\mathcal{R}(a_t) = \{\, a_j \in \mathcal{C}_\tau(a_t) \mid \phi(a_t,a_j) = Pass \,\}.
$$

The paper characterizes this as a recall-first, precision-second design: embedding similarity proposes candidate redundancies, and LLM verification determines factual equivalence [2604.19047].

## 3. Corpora, overlap profile, and domain scope

RedQA is built on four corpora: **Finance**, **Legal**, **Patent**, and **General-Wiki**. The first three are the target enterprise-style domains, while General-Wiki serves as a low-overlap baseline [2604.19047].

Finance uses **SEC 10-K reports from 2023–2024**, including Apple, Microsoft, Amazon, Meta, Tesla, and NVIDIA. Legal uses the **2023 U.S. Code**, restricted to six titles. Patent uses **USPTO documents from 2023–2024** across five technical areas. General-Wiki uses **HotpotQA Wikipedia passages** as the low-redundancy comparison set. The enterprise corpora were chosen because they naturally exhibit repeated boilerplate language, standardized phrasing, cross-references, and near-duplicate factual content.

The paper reports the following redundancy and similarity profile:

| Corpus | Redundancy (%) | Similarity (%) |
|---|---:|---:|
| Finance | 63.2 | 35.1 |
| Patent | 49.7 | 29.0 |
| Legal | 25.1 | 40.7 |
| General-Wiki | 1.4 | 8.8 |

The associated corpus sizes are **3,281 chunks** and **645 samples** for Finance, **3,339 chunks** and **480 samples** for Patent, **4,090 chunks** and **605 samples** for Legal, and **3,218 chunks** and **688 samples** for General-Wiki [2604.19047].

The benchmark formalizes both corpus-level similarity and atomic-level redundancy. Similarity is defined at chunk level as mean pairwise cosine similarity:

$$
\mathrm{Sim} = \mathbb{E}_{i<j}\left[\cos\!\left(\mathbf{e}(p_i), \mathbf{e}(p_j)\right)\right].
$$

Redundancy is defined at atomic level as

$$
\mathrm{Red} = \frac{|\{a \in \mathcal{T} : \exists\, a' \notin \mathrm{chunk}(a)\ \text{s.t.}\ a \equiv a' \}|}{|\mathcal{T}|}.
$$

These two quantities are not treated as interchangeable. The reported corpus statistics make that distinction explicit: Finance is the most redundant, Legal has the highest similarity, and General-Wiki is both low-similarity and low-redundancy [2604.19047].

## 4. Question generation, logical filtering, and metrics

Questions in RedQA are generated from selected atomic facts. The procedure samples \(N\) atomic units, selects \(M\) connected units as evidence, generates \(S\) paraphrased candidate questions, filters them with hard logical checks, and ranks the survivors via CRRF. The benchmark supports **1–4 hop** questions [2604.19047].

Logical filtering is zero-tolerance. Each candidate question must satisfy five criteria:

- **Contextual Independence**
- **Answer Exclusion**
- **Information Equivalence**
- **Question Clarity**
- **Answerability**

Only candidates that pass every criterion are retained:

$$
Q^{\ast} = \{\, q \in Q \mid \forall \ell \in \mathcal{L},\; \phi_\ell(q) = Pass \,\}.
$$

Remaining questions are ranked by four criteria: **Connectivity**, **Fluency**, **Essentiality**, and **Validity**. The highest-ranked question is selected [2604.19047].

Evaluation is explicitly redundancy-aware. The primary retrieval metrics are **Coverage@10** and **PerfRecall@10**. Coverage@10 measures the fraction of required information retrieved in the top 10, while PerfRecall@10 is a binary success metric that equals 1 only if **all required information** appears in the top 10. Required information is defined at the **chunk level**, and the gold labels are redundancy-aware rather than canonical-passage-only. The appendix additionally reports **NDCG@10** and **MRR** [2604.19047].

The filtering stages were validated by human evaluation on **480 items**, judged by **eight NLP researchers**. The reported aggregate outcome is a recall-oriented system with only **4.2% false negatives** overall. For **Information Filtering**, the paper reports **precision 57.5%**, **recall 89.6%**, and **accuracy 75.4%**. For **Question Filtering**, it reports **precision 50.8%**, **recall 83.6%**, and **accuracy 70.4%** [2604.19047].

## 5. Empirical findings and difficulty profile

The central empirical result is that RedQA is substantially harder than low-overlap benchmarks. For **Qwen3-8B** on **4-hop** retrieval, **PerfRecall@10** is reported as **66.4%** on General-Wiki, but only **8.5%** on Finance, **27.9%** on Patent, and **5.0%** on Legal [2604.19047]. This contrast is the paper’s principal evidence that standard benchmark performance can overestimate real-world retrieval robustness.

Performance also degrades sharply with hop depth. The paper gives Finance as a concrete example: **PerfRecall@10** drops from **90.1%** at **1-hop** to **8.5%** at **4-hop**, a decline of **81.6 points** [2604.19047]. The interpretation offered is that high-overlap corpora cause top-\(k\) retrieval to concentrate on redundant evidence rather than complementary evidence, so compositional coverage collapses as the number of required facts increases.

A further claim is that **similarity matters more than redundancy** for retrieval difficulty. Across the enterprise corpora, Legal has the highest similarity and lowest redundancy, Finance has the highest redundancy and moderate similarity, and Patent has the lowest similarity among the enterprise domains with intermediate redundancy. For **Qwen3-8B**, the paper reports **PerfRecall@10 = 41.49%** for Legal, **47.44%** for Finance, and **63.12%** for Patent, and interprets this pattern as evidence that similarity drives retrieval confusion, while redundancy can sometimes help by providing alternative evidence paths [2604.19047].

Strong dense retrievers still underperform in these domains. The main table reports **OpenAI-Large PerfRecall@10 = 86.48%** on General-Wiki, compared with **42.95%** on Finance, **37.36%** on Legal, and **61.46%** on Patent [2604.19047]. The paper also notes that LLM-based embeddings such as Qwen3 and E5-Mistral generally outperform smaller embedding-only baselines, but do not solve the high-overlap problem.

The benchmark also includes end-to-end RAG experiments with **GPT-5 Mini** as generator. Reported end-to-end accuracy is **80.81%** on General-Wiki, **55.85%** on Finance, **45.82%** on Legal, and **58.40%** on Patent [2604.19047]. The authors’ conclusion is that retrieval quality is the dominant factor, particularly in enterprise corpora where parametric knowledge overlaps less with the evaluation domain.

## 6. Position within QA evaluation research, implications, and limitations

RedQA occupies a specific niche within retrieval-augmented QA evaluation. It differs from **Total Recall QA**, where accurate answer generation requires retrieving **all relevant documents** and synthesizing across **all relevant documents** to compute a single answer; RedQA instead formalizes retrieval sufficiency under redundancy, explicitly allowing alternative passages that contain semantically equivalent facts [2603.18516][2604.19047]. The contrast is methodological rather than oppositional: TRQA stresses exhaustive recovery of the full relevant set, whereas RedQA stresses robustness when the same fact is distributed across multiple highly similar documents.

Relative to system papers such as **QUASAR**, which uses question understanding, evidence retrieval, reranking or filtering, and answer generation over text, tables, and knowledge graphs, or **RE-RAG**, which introduces a relevance estimator that provides both ranking and confidence for answerability, RedQA functions primarily as an evaluation resource rather than as an answering architecture [2412.07420][2406.05794]. A plausible implication is that RedQA is especially suited to evaluating pipelines that decompose retrieval, reranking, and generation, because its metrics isolate whether a system has assembled a sufficient evidence set before generation begins. The same implication extends to earlier ranker-reader work such as **R\(^3\)**, which learned passage usefulness for open-domain QA but operated in a setting where passage identity rather than redundancy-aware sufficiency was the dominant evaluation lens [1709.00023].

The benchmark’s practical message is explicit: enterprise RAG should not be evaluated only on low-overlap QA datasets. The paper argues instead for benchmarks that track **fact-level redundancy**, include **semantically equivalent evidence**, and measure whether retrieved context is **sufficient**, not merely identical to a canonical annotation [2604.19047]. This aligns with a broader movement toward domain- and corpus-specific benchmarking, including documentation QA benchmarks such as **ORD-QA**, which ground answers in specific reference chunks for EDA documentation [2407.15353].

The paper also states several limitations. RedQA relies on **LLM judges**, so it inherits model biases and calibration issues. Redundancy detection uses a **fixed similarity threshold** \(\tau = 0.5\), which may not generalize optimally. **CRRF** is validated primarily on **sentence ranking in Wikipedia passages**. Some high-hop questions can become somewhat **list-like**, and retriever comparisons do not fully control for differences in training data across embedding families [2604.19047]. These caveats do not alter the benchmark’s central contribution, but they delimit the scope of its claims.

In the benchmark literature, RedQA’s distinctive contribution is therefore not simply another set of questions. It is a change in what is counted as correct retrieval: not recovery of one canonical support passage, and not exhaustive recovery of every relevant document, but recovery of a top-\(k\) evidence set that is collectively sufficient in a corpus where redundancy and similarity are first-order properties of the deployment environment [2604.19047].

Source: https://www.emergentmind.com/topics/redqa