Enterprise RAG Benchmark

• Enterprise RAG Benchmark is a framework for evaluating retrieval-augmented generation on structured, heterogeneous enterprise corpora with multi-modal, multi-document, and hierarchical challenges.
• It assesses systems on tasks like contradiction handling, provenance tracking, and human-in-the-loop review, ensuring both operational performance and diagnostic insights.
• Empirical findings highlight improvements with specialized retrieval methods while emphasizing persistent trade-offs in accuracy, latency, and reproducibility.

Enterprise RAG Benchmark denotes a class of evaluation settings for retrieval-augmented generation systems operating over enterprise or enterprise-like corpora rather than open-web QA collections. In this literature, the benchmark target is not limited to end-to-end answer accuracy: it also includes multi-document synthesis, hierarchical and version-aware retrieval, contradiction handling, multimodal evidence use, abstention on missing information, provenance, and operational evaluation under realistic source heterogeneity (Sun et al., 5 May 2026, Cohen et al., 13 May 2025, Narita et al., 3 Apr 2026).

1. Definition and positioning

Enterprise benchmarking emerged from the observation that public QA and retrieval suites do not adequately capture fixed knowledge-base snapshots, proprietary corpora, compliance requirements, black-box deployment constraints, or the novelty of real user questions in production support and knowledge workflows. Some works therefore foreground enterprise-grade security, auditability, and integration readiness; others emphasize human-in-the-loop review for novel questions, black-box REST-level comparison of deployed RAG stacks, or explainable decomposition of retrieval and generation failures (Bruckhaus, 2024, Packowski et al., 2024, Rengo et al., 10 Apr 2025, Friel et al., 2024).

Benchmark family	Corpus or environment	Primary stressor
CFR benchmark	Code of Federal Regulations	Hierarchy, cross-references, supersession
WixQA	Wix Help Center snapshot	Multi-article procedural support QA
EnterpriseRAG-Bench	Synthetic multi-app internal corpus	Noise, cross-document coherence, constrained retrieval
HERB	Heterogeneous enterprise artifacts	Source-aware multi-hop deep search
MTRAG-UN	Multi-turn enterprise-style conversations	Unanswerable, underspecified, non-standalone turns
RAGBench	Multi-domain industry corpora	Explainable retrieval/generation diagnostics

This benchmark family spans fixed public KB snapshots, synthetic internal corpora, workflow-grounded heterogeneous artifacts, multi-turn conversational settings, and evaluation harnesses that operate directly against deployed endpoints (Cohen et al., 13 May 2025, Sun et al., 5 May 2026, Choubey et al., 29 Jun 2025, Rosenthal et al., 26 Feb 2026).

2. Corpora and workload design

Enterprise RAG benchmarks are distinguished by corpus structure. The CFR benchmark uses structured XML from the Code of Federal Regulations, whose strict nesting runs from Title to Paragraph and whose text exhibits dense explicit citations and annual versioning; the paper presents this as a stress test for superseding logic, multi-hop references, and temporal validity. WixQA instead releases a fixed snapshot of 6,221 English-only Wix Help Center articles and grounds three QA datasets in that snapshot: 200 expert-written questions, 200 expert-validated simulated QA pairs, and 6,221 synthetic one-per-article pairs (Chakraborty et al., 14 Apr 2026, Cohen et al., 13 May 2025).

Other benchmarks push closer to internal enterprise data. EnterpriseRAG-Bench contains 511,962 synthetic documents across nine source types—Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence—with realistic volume skew, misfiling, near-duplicates, conflicting information, and cross-document coherence grounded in shared projects, people, and initiatives. HERB assembles a retrieval pool of 39,190 artifacts spanning 33,632 Slack messages, 321 meeting transcripts, 400 internal documents, 3,562 GitHub PRs, 575 URLs, 530 employee profiles, and 120 customer profiles, distributed across 30 products and 6 organizations (Sun et al., 5 May 2026, Choubey et al., 29 Jun 2025).

Conversational and diagnostic benchmarks define the workload differently. MTRAG-UN releases 666 tasks derived from over 2,800 conversation turns across six domains, including Banking and Telco corpora with 4,497 and 4,616 documents respectively. The diagnostic framework of "Overcoming the 'Impracticality' of RAG" uses 34 public enterprise-relevant documents totaling 1,699 pages and 100 queries, chunked by page, to expose retrieval, reasoning, structural, and explainability failures in real-world materials such as financial reports, technical specifications, and regulatory documents (Rosenthal et al., 26 Feb 2026, Narita et al., 3 Apr 2026).

3. Enterprise-specific difficulty structure

A central contribution of this literature is the claim that enterprise difficulty is composite rather than monolithic. The four-axis taxonomy in (Narita et al., 3 Apr 2026) decomposes benchmark difficulty into Reasoning Complexity, Retrieval Difficulty, Source Structure and Modality, and Explainability Requirement. Each axis is annotated with Boolean elements, and high-difficulty subsets are derived by majority rule; the resulting D-values report accuracy restricted to queries that are difficult along a specific axis.

Several benchmarks instantiate these abstractions through concrete task types. In the CFR setting, systems must follow explicit REFERS_TO citations, enforce temporal precedence through SUPERSEDES, and preserve hierarchical context through CONTAINS; query examples require year-specific definitions, superseding clause resolution, and multi-hop reference chains (Chakraborty et al., 14 Apr 2026). EnterpriseRAG-Bench operationalizes related pressures through ten categories: Basic, Semantic, Intra-Document Reasoning, Project Related, Constrained, Conflicting Info, Completeness, Miscellaneous, High Level, and Info Not Found (Sun et al., 5 May 2026).

Multi-turn and conflict-centric benchmarks extend the taxonomy further. MTRAG-UN isolates UNanswerable, UNderspecified, NONstandalone, and UNclear turns, thereby stressing abstention, clarification, contextual query rewriting, and self-correction (Rosenthal et al., 26 Feb 2026). ContraGen defines six contradiction types—Temporal, Numerical, Authority, Process, Policy Reversal, and Specificity—and evaluates both self-contradictions and pairwise cross-document contradictions in enterprise-style documents (Mantravadi et al., 3 Oct 2025). VisualRAG adds cross-modal enterprise document understanding, targeting corpora in which critical instructions reside in screenshots, diagrams, captions, and OCR text rather than plain text alone (Mannam et al., 19 Jun 2025). HERB frames the same broad problem as "Deep Search": source-aware, multi-hop reasoning across heterogeneous enterprise systems, where evidence chains cross Slack, meetings, documents, PR metadata, URLs, and identity tables (Choubey et al., 29 Jun 2025). This suggests that "enterprise difficulty" is increasingly treated as a joint property of source structure, temporal state, interaction history, and evidence modality rather than of query semantics alone.

4. Metrics, protocols, and evaluation logic

The metric landscape is correspondingly heterogeneous. The CFR benchmark reports categorical outcomes per query—Correct/Complete, Incomplete/Inaccurate, and Refusals/No Answer—and supplements them with an Overlap Coefficient over answer and source keywords. WixQA evaluates token F1, BLEU, ROUGE-1, ROUGE-2, and two GPT-4o judge metrics, Factuality and Context Recall. Corpus2Skill inherits the same quality metrics on WixQA and standardizes the LLM judge input by taking the last five retrieved documents or passages, concatenating them, and truncating to an 8,000-character window (Chakraborty et al., 14 Apr 2026, Cohen et al., 13 May 2025, Sun et al., 16 Apr 2026).

Explainable evaluation is formalized most explicitly by TRACe in RAGBench. TRACe decomposes evaluation into Context Relevance, Context Utilization, Completeness, and Adherence. Relevance measures how much of retrieved context is actually relevant to the question; Utilization measures how much of the context the model uses; Completeness measures overlap between relevant and utilized spans; Adherence is an example-level boolean requiring that all response content be supported by the provided context. The benchmark provides sentence-level relevance and utilization spans and per-response-sentence support annotations, making failure attribution directly inspectable (Friel et al., 2024).

Other frameworks target diagnostic isolation rather than span explainability. The four-axis benchmark defines Dimensional Diagnostic Accuracy as

$$\mathrm{D}_i = \frac{1}{|Q_i|} \sum_{q \in Q_i} c(q),$$

where $Q_i$ is the subset of queries labeled high difficulty on axis $i$ and $c(q)$ is binary correctness (Narita et al., 3 Apr 2026). The case-aware LLM-as-a-judge framework for enterprise multi-turn RAG evaluates eight turn-level metrics—Hallucination, Retrieval Correctness, Context Sufficiency, Answer Helpfulness, Answer Type Fit, Identifier Integrity, Case Issue Identification, and Resolution Alignment—using deterministic prompts, severity bands, strict JSON outputs, and a weighted aggregate $S_{\text{final}}$ (Chhabra et al., 23 Feb 2026). For long-form research-style enterprise tasks, ADORE employs the RACE framework, which scores Readability, Instruction Following, Comprehensiveness, and Insight with task-specific weights (You et al., 26 Jan 2026).

Recent work also challenges rank-centric IR evaluation directly. "Practical RAG Evaluation" argues that RAG is a set-consumption rather than list-browsing setting and introduces RA-nWG@K, together with pool-restricted oracle ceilings (PROC) and %PROC, to separate retrieval headroom from ordering headroom under a cost-latency-quality lens (Dallaire, 12 Nov 2025). SCARF occupies a different layer: it is a modular black-box harness that benchmarks deployed RAG systems via REST adapters, automates ingest-query-score-export loops, and supports optional LLM-as-a-judge evaluation without prescribing a single enterprise metric schema (Rengo et al., 10 Apr 2025).

5. Systems and empirical findings

The empirical record across these benchmarks is dominated by structural retrieval failures. In the CFR benchmark, standard vector-based RAG achieved 5 Correct/Complete answers out of 20 questions (25%), whereas KG-RAG achieved 19 out of 20 (95%), a reported 70 percentage-point improvement; for the question on the 2002 definition of “account” in CFR 561.2, the Overlap Coefficient was 57.14% for KG-RAG and 9.52% for vector RAG (Chakraborty et al., 14 Apr 2026). The paper attributes vector failures to retrieval of semantically similar but outdated clauses, failure to traverse cited dependencies, and contextual fragmentation.

On WixQA, Corpus2Skill outperformed dense retrieval, RAPTOR, and an agentic baseline across all reported quality metrics. Its reported scores were F1 0.460, BLEU 0.137, ROUGE-1 0.476, ROUGE-2 0.231, Factuality 0.729, and Context Recall 0.652, at a cost of 53,487 input tokens and \$0.172 per query. The paper interprets this as evidence that a visible hierarchical corpus map and serve-time navigation improve evidence gathering beyond flat retrieval (Sun et al., 16 Apr 2026). WixQA itself showed that dense E5 retrieval improved Context Recall over BM25 on realistic multi-article support queries, while Synthetic single-article questions were materially easier than ExpertWritten and Simulated sets (Cohen et al., 13 May 2025).

EnterpriseRAG-Bench produced a different result profile. With top-10 retrieval and GPT-5.4 Medium Reasoning generation, BM25 led on overall correctness at 68.8%, vector search achieved 51.4%, and a Bash Agent achieved 60.6% correctness but the highest completeness at 61.1% and the fewest Invalid Extra Documents at 2.0. Completeness questions were the hardest, and scaling analysis showed that as corpus size increased from 5k to 512k documents, local cosine similarity rose and Recall@10 declined for both BM25 and vector retrieval (Sun et al., 5 May 2026).

HERB makes retrieval bottlenecks even more explicit. In full-RAG evaluation over the entire 39,190-item pool, the best standard baseline was Hybrid at an average score of 20.61, while the best agentic configuration, ReAct with GPT-4o, reached 32.96. However, under product-specific long-context evaluation without retrieval, Gemini-2.5-Flash reached 76.55 versus 41.86 for product-specific RAG, and with oracle evidence it reached 85.76. The paper therefore identifies retrieval—not final generation—as the dominant bottleneck in source-aware multi-hop enterprise search (Choubey et al., 29 Jun 2025).

MTRAG-UN reports the same pattern in conversational form. Elser was the strongest retriever, with Recall@5 improving from 0.40 under last-turn retrieval to 0.49 under query rewriting, and non-standalone turns benefited especially strongly from rewriting. On the generation side, GPT-OSS-120B was the strongest model, but the paper still reports a large gap between reference-passage and RAG settings, and continued difficulty on underspecified turns and clarification requests (Rosenthal et al., 26 Feb 2026).

Several specialized benchmarks show similar relative trends. In ContraGen, the confidence-weighted hybrid detector outperformed both NLI and GPT-4o-only judgment, reaching 90.1% accuracy and 87.7% F1 on 141 self-contradiction pairs and 87.8% accuracy and 64.9% F1 on 426 pairwise contradiction pairs (Mantravadi et al., 3 Oct 2025). In RAGBench, a fine-tuned DeBERTa-v3-Large evaluator generally outperformed GPT-3.5, RAGAS, and TruLens on hallucination detection; for example, Hallucination AUROC on ExpertQA was 0.87 for DeBERTa versus 0.55 for GPT-3.5 (Friel et al., 2024). VisualRAG reports that a final multimodal weighting of 30% text, 15% image, 25% caption, and 30% OCR improved performance by 57.3% over a text-only baseline (Mannam et al., 19 Jun 2025). For long-form enterprise synthesis, ADORE reports rank 1 on DeepResearch Bench with Overall RACE 52.65 and a 77.21% win rate on DeepConsult (You et al., 26 Jan 2026).

6. Reproducibility, limitations, and emerging directions

A recurrent feature of enterprise benchmarking is partial reproducibility. The CFR benchmark provides illustrative Python snippets and public CFR data, but not benchmark splits, evaluation scripts, or significance testing; its 20-query set is curated by the authors, and no train/dev/test partition is specified (Chakraborty et al., 14 Apr 2026). Corpus2Skill provides prompts, traces, and parameter settings, but compiled skill trees are not distributed, incremental updates are not supported, and serve-time behavior depends on Anthropic Skills API limits such as 8 skills per request and 30 MB per skill (Sun et al., 16 Apr 2026).

Synthetic corpora solve privacy constraints but introduce representational trade-offs. EnterpriseRAG-Bench explicitly notes a synthetic-versus-real gap, flattened JSON representations, correction-aware but revisable gold sets, and weaker cross-document coherence outside high-fidelity clusters (Sun et al., 5 May 2026). The four-axis diagnostic benchmark likewise omits cost and latency from its current evaluation, does not model interactions among axes, and uses only 100 queries, which the paper treats as sufficient to demonstrate diagnostic capability rather than broad statistical generalization (Narita et al., 3 Apr 2026).

At the framework level, SCARF exposes another limitation: it supports black-box, multi-configuration testing of deployed RAG systems, but the current paper does not include built-in latency, throughput, cost tracking, or statistical testing (Rengo et al., 10 Apr 2025). Recent work therefore turns to benchmark generation and metric redesign. IRB automates factuality benchmark creation through a factual scaffold and an algorithmic scaffold, using grounded citation evidence to produce refreshable benchmarks and arguing that improving retrieval may yield more cost-effective gains than scaling the generator (Do et al., 8 Feb 2026). The rarity-aware set metric RA-nWG@K, together with PROC and %PROC, reframes evaluation around whether the top-$K$ set contains decisive evidence under budget and latency constraints rather than around classical rank smoothness (Dallaire, 12 Nov 2025).

A parallel line of work argues that enterprise benchmarking cannot be fully automated. The content-design perspective in (Packowski et al., 2024) reports that exact-match-style metrics, cosine similarity, BLEU, ROUGE, METEOR, BERTScore, and LLM-as-judge methods “have not been useful” for evaluating novel user questions from real users, motivating a human-in-the-lead workflow centered on judgments such as Valid question, Article exists, Search success, and Good answer. Taken together, these papers suggest that an enterprise benchmark is best understood not as a single dataset, but as a composite evaluation regime: corpus-grounded, provenance-sensitive, structurally diagnostic, and explicitly aware of operational trade-offs, benchmark refresh, and human review.