RedQA: Redundancy-Aware QA Benchmark

• RedQA is a redundancy-aware benchmark that redefines QA evaluation by measuring evidence sufficiency across near-duplicate passages.
• It employs the RARE framework to decompose documents into atomic facts and systematically track redundancy using embedding similarity and LLM verification.
• Empirical findings reveal that standard QA systems overestimate retrieval quality in enterprise domains due to overlap-induced evidence bottlenecks.

RedQA is a redundancy-aware retrieval and question-answering benchmark produced by the RARE framework for high-similarity corpora such as financial filings, legal codes, and patents. It was created to evaluate retrieval-augmented generation under conditions where the same fact may appear in many near-duplicate passages, so a retriever may return a passage different from the annotated source chunk yet still provide sufficient evidence. Unlike standard benchmarks that assume distinct documents, low overlap, and one canonical supporting passage, RedQA labels semantically equivalent evidence at the atomic-fact level and evaluates sufficiency rather than exact passage match (Cho et al., 21 Apr 2026).

1. Motivation and benchmark philosophy

RedQA is motivated by a mismatch between standard QA evaluation and enterprise-style retrieval environments. The underlying claim is that most conventional QA and retrieval benchmarks assume distinct passages or documents, low overlap between evidence sources, and a single correct supporting passage. In high-similarity corpora, that assumption becomes brittle: the same fact may recur across multiple passages, retrievers may return a different but equally valid passage than the canonical annotation, and standard evaluation can therefore underestimate retrieval quality. The paper also argues the converse: systems that perform well on low-overlap benchmarks may generalize poorly to corpora with strong inter-document similarity and redundancy (Cho et al., 21 Apr 2026).

The benchmark therefore redefines what counts as relevant evidence. If a question is generated from a source chunk $P_i$, the answer may also appear in other chunks $P_j$, $j \neq i$. Under RedQA, retrieving $P_j$ is still correct if it supports the answer. This design shifts evaluation from exact passage identity to evidence sufficiency. In practical terms, the benchmark asks whether the retrieved set contains the information required to answer the question, not whether it reproduces a single annotated provenance path.

This design choice is especially consequential for multi-hop retrieval. In a redundant corpus, top-$k$ retrieval can be dominated by passages that repeat the same fact, leaving complementary facts unretrieved. The paper describes this as an overlap-induced evidence bottleneck: redundancy may provide alternative evidence paths, but high similarity can also crowd out the diverse evidence required for compositional answering (Cho et al., 21 Apr 2026).

2. RARE framework and atomic-fact construction

RedQA is instantiated through RARE, short for Redundancy-Aware Retrieval Evaluation, which constructs the benchmark in three stages: Valid Information Selection, Systematic Redundancy Tracking, and Question–Answer Generation (Cho et al., 21 Apr 2026).

A central operation is document decomposition into atomic information units. Each input chunk is decomposed by an LLM into atomic facts that are defined as single-claim, self-contained, suitable for fine-grained redundancy tracking, and useful as building blocks for multi-hop question generation. These atoms are then filtered by a binary LLM judge $\phi(\cdot)\in\{Pass,Fail\}$. A valid atom must be complete and self-contained, non-trivial and utility-relevant, and factual rather than metadata or document structure.

Valid atoms are ranked by five criteria: Validity, Completeness, Specificity, Clarity, and Questionability. To improve multi-criteria LLM judgment, the framework introduces CRRF, or Criterion-wise Prompting with Reciprocal Rank Fusion. CRRF evaluates each criterion independently and fuses the resulting ranks by

$$s(x) = \sum_{i=1}^{N} \frac{1}{\mathrm{rank}_i(x)}.$$

The stated rationale is that separate prompting reduces cross-criterion interference, while rank-based fusion is more robust than averaging raw confidence scores. In ablation, Separate + RRF is reported as the best-performing setup, achieving NDCG@3 = 0.463 on GPT-5 Nano and NDCG@3 = 0.467 on GPT-5, and it is described as more stable across runs than joint prompting (Cho et al., 21 Apr 2026).

Systematic redundancy tracking proceeds in two stages. For each target atomic fact $a_t$, the framework first retrieves high-recall candidates by embedding similarity:

$$\mathcal{C}_\tau(a_t) = \bigl\{\, a_j \in \mathcal{A} \ \bigm|\ \mathrm{chunk}(a_j)\neq \mathrm{chunk}(a_t),\ \text{sim}(a_t,a_j) \ge \tau \,\bigr\},$$

with a recall-oriented threshold $\tau = 0.5$. It then refines this set through an LLM verifier:

$P_j$0

The paper characterizes this as a recall-first, precision-second design: embedding similarity proposes candidate redundancies, and LLM verification determines factual equivalence (Cho et al., 21 Apr 2026).

3. Corpora, overlap profile, and domain scope

RedQA is built on four corpora: Finance, Legal, Patent, and General-Wiki. The first three are the target enterprise-style domains, while General-Wiki serves as a low-overlap baseline (Cho et al., 21 Apr 2026).

Finance uses SEC 10-K reports from 2023–2024, including Apple, Microsoft, Amazon, Meta, Tesla, and NVIDIA. Legal uses the 2023 U.S. Code, restricted to six titles. Patent uses USPTO documents from 2023–2024 across five technical areas. General-Wiki uses HotpotQA Wikipedia passages as the low-redundancy comparison set. The enterprise corpora were chosen because they naturally exhibit repeated boilerplate language, standardized phrasing, cross-references, and near-duplicate factual content.

The paper reports the following redundancy and similarity profile:

Corpus	Redundancy (%)	Similarity (%)
Finance	63.2	35.1
Patent	49.7	29.0
Legal	25.1	40.7
General-Wiki	1.4	8.8

The associated corpus sizes are 3,281 chunks and 645 samples for Finance, 3,339 chunks and 480 samples for Patent, 4,090 chunks and 605 samples for Legal, and 3,218 chunks and 688 samples for General-Wiki (Cho et al., 21 Apr 2026).

The benchmark formalizes both corpus-level similarity and atomic-level redundancy. Similarity is defined at chunk level as mean pairwise cosine similarity:

$P_j$1

Redundancy is defined at atomic level as

$P_j$2

These two quantities are not treated as interchangeable. The reported corpus statistics make that distinction explicit: Finance is the most redundant, Legal has the highest similarity, and General-Wiki is both low-similarity and low-redundancy (Cho et al., 21 Apr 2026).

4. Question generation, logical filtering, and metrics

Questions in RedQA are generated from selected atomic facts. The procedure samples $P_j$3 atomic units, selects $P_j$4 connected units as evidence, generates $P_j$5 paraphrased candidate questions, filters them with hard logical checks, and ranks the survivors via CRRF. The benchmark supports 1–4 hop questions (Cho et al., 21 Apr 2026).

Logical filtering is zero-tolerance. Each candidate question must satisfy five criteria:

• Contextual Independence
• Answer Exclusion
• Information Equivalence
• Question Clarity
• Answerability

Only candidates that pass every criterion are retained:

$P_j$6

Remaining questions are ranked by four criteria: Connectivity, Fluency, Essentiality, and Validity. The highest-ranked question is selected (Cho et al., 21 Apr 2026).

Evaluation is explicitly redundancy-aware. The primary retrieval metrics are Coverage@10 and PerfRecall@10. Coverage@10 measures the fraction of required information retrieved in the top 10, while PerfRecall@10 is a binary success metric that equals 1 only if all required information appears in the top 10. Required information is defined at the chunk level, and the gold labels are redundancy-aware rather than canonical-passage-only. The appendix additionally reports NDCG@10 and MRR (Cho et al., 21 Apr 2026).

The filtering stages were validated by human evaluation on 480 items, judged by eight NLP researchers. The reported aggregate outcome is a recall-oriented system with only 4.2% false negatives overall. For Information Filtering, the paper reports precision 57.5%, recall 89.6%, and accuracy 75.4%. For Question Filtering, it reports precision 50.8%, recall 83.6%, and accuracy 70.4% (Cho et al., 21 Apr 2026).

5. Empirical findings and difficulty profile

The central empirical result is that RedQA is substantially harder than low-overlap benchmarks. For Qwen3-8B on 4-hop retrieval, PerfRecall@10 is reported as 66.4% on General-Wiki, but only 8.5% on Finance, 27.9% on Patent, and 5.0% on Legal (Cho et al., 21 Apr 2026). This contrast is the paper’s principal evidence that standard benchmark performance can overestimate real-world retrieval robustness.

Performance also degrades sharply with hop depth. The paper gives Finance as a concrete example: PerfRecall@10 drops from 90.1% at 1-hop to 8.5% at 4-hop, a decline of 81.6 points (Cho et al., 21 Apr 2026). The interpretation offered is that high-overlap corpora cause top-$P_j$7 retrieval to concentrate on redundant evidence rather than complementary evidence, so compositional coverage collapses as the number of required facts increases.

A further claim is that similarity matters more than redundancy for retrieval difficulty. Across the enterprise corpora, Legal has the highest similarity and lowest redundancy, Finance has the highest redundancy and moderate similarity, and Patent has the lowest similarity among the enterprise domains with intermediate redundancy. For Qwen3-8B, the paper reports PerfRecall@10 = 41.49% for Legal, 47.44% for Finance, and 63.12% for Patent, and interprets this pattern as evidence that similarity drives retrieval confusion, while redundancy can sometimes help by providing alternative evidence paths (Cho et al., 21 Apr 2026).

Strong dense retrievers still underperform in these domains. The main table reports OpenAI-Large PerfRecall@10 = 86.48% on General-Wiki, compared with 42.95% on Finance, 37.36% on Legal, and 61.46% on Patent (Cho et al., 21 Apr 2026). The paper also notes that LLM-based embeddings such as Qwen3 and E5-Mistral generally outperform smaller embedding-only baselines, but do not solve the high-overlap problem.

The benchmark also includes end-to-end RAG experiments with GPT-5 Mini as generator. Reported end-to-end accuracy is 80.81% on General-Wiki, 55.85% on Finance, 45.82% on Legal, and 58.40% on Patent (Cho et al., 21 Apr 2026). The authors’ conclusion is that retrieval quality is the dominant factor, particularly in enterprise corpora where parametric knowledge overlaps less with the evaluation domain.

6. Position within QA evaluation research, implications, and limitations

RedQA occupies a specific niche within retrieval-augmented QA evaluation. It differs from Total Recall QA, where accurate answer generation requires retrieving all relevant documents and synthesizing across all relevant documents to compute a single answer; RedQA instead formalizes retrieval sufficiency under redundancy, explicitly allowing alternative passages that contain semantically equivalent facts (Rafiee et al., 19 Mar 2026, Cho et al., 21 Apr 2026). The contrast is methodological rather than oppositional: TRQA stresses exhaustive recovery of the full relevant set, whereas RedQA stresses robustness when the same fact is distributed across multiple highly similar documents.

Relative to system papers such as QUASAR, which uses question understanding, evidence retrieval, reranking or filtering, and answer generation over text, tables, and knowledge graphs, or RE-RAG, which introduces a relevance estimator that provides both ranking and confidence for answerability, RedQA functions primarily as an evaluation resource rather than as an answering architecture (Christmann et al., 2024, Kim et al., 2024). A plausible implication is that RedQA is especially suited to evaluating pipelines that decompose retrieval, reranking, and generation, because its metrics isolate whether a system has assembled a sufficient evidence set before generation begins. The same implication extends to earlier ranker-reader work such as R$P_j$8, which learned passage usefulness for open-domain QA but operated in a setting where passage identity rather than redundancy-aware sufficiency was the dominant evaluation lens (Wang et al., 2017).

The benchmark’s practical message is explicit: enterprise RAG should not be evaluated only on low-overlap QA datasets. The paper argues instead for benchmarks that track fact-level redundancy, include semantically equivalent evidence, and measure whether retrieved context is sufficient, not merely identical to a canonical annotation (Cho et al., 21 Apr 2026). This aligns with a broader movement toward domain- and corpus-specific benchmarking, including documentation QA benchmarks such as ORD-QA, which ground answers in specific reference chunks for EDA documentation (Pu et al., 2024).

The paper also states several limitations. RedQA relies on LLM judges, so it inherits model biases and calibration issues. Redundancy detection uses a fixed similarity threshold $P_j$9, which may not generalize optimally. CRRF is validated primarily on sentence ranking in Wikipedia passages. Some high-hop questions can become somewhat list-like, and retriever comparisons do not fully control for differences in training data across embedding families (Cho et al., 21 Apr 2026). These caveats do not alter the benchmark’s central contribution, but they delimit the scope of its claims.

In the benchmark literature, RedQA’s distinctive contribution is therefore not simply another set of questions. It is a change in what is counted as correct retrieval: not recovery of one canonical support passage, and not exhaustive recovery of every relevant document, but recovery of a top-$j \neq i$0 evidence set that is collectively sufficient in a corpus where redundancy and similarity are first-order properties of the deployment environment (Cho et al., 21 Apr 2026).