EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge

Abstract: Retrieval-Augmented Generation (RAG) has become the standard approach for grounding LLMs in information that was not available during training. While existing datasets and benchmarks focus on web or other public sources, there is still no widely adopted dataset that realistically reflects the nature of company-internal knowledge. Meanwhile, startups, enterprises, and researchers are increasingly developing AI Agents designed to operate over exactly this kind of proprietary data. To close this gap, we release a synthetic enterprise corpus, its generation framework, and a leaderboard. We present EnterpriseRAG-Bench, a dataset consisting of approximately 500,000 documents spanning nine enterprise source types (Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence) and 500 questions across ten categories that test distinct retrieval and reasoning capabilities. The corpus is generated with cross-document coherence (grounded in shared projects, people, and initiatives) and augmented with realistic noise such as misfiled documents, near-duplicates, and conflicting information. The question set ranges from simple single-document lookups to multi-document reasoning, constrained retrieval, conflict resolution, and recognizing when information is absent. The generation framework lets teams generate variants tailored to their own industry, scale, and source mix. The dataset, code, evaluation harness, and leaderboard are available at https://github.com/onyx-dot-app/EnterpriseRAG-Bench.

• The paper introduces EnterpriseRAG-Bench, a synthetic benchmark for evaluating retrieval-augmented generation on company internal data.
• It employs a systematic generation pipeline to create 500k documents from nine diverse enterprise sources, capturing real-world noise and organizational structure.
• Empirical results reveal BM25 outperforming dense vector retrieval, highlighting the challenge of applying public embedding models to proprietary corpora.

EnterpriseRAG-Bench: A Benchmark for Retrieval-Augmented Generation on Company Internal Knowledge

Motivation and Problem Statement

Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for grounding LLMs in up-to-date and domain-specific information. Existing benchmarks like Natural Questions, MS MARCO, and BEIR predominantly assess retrieval over public, web-scale corpora. However, a critical and operationally vital use case for RAG is the augmentation of LLMs with company-internal knowledge, encompassing emails, tickets, collaborative documents, chats, and records housed in disparate, often noisy, enterprise systems. The lack of a realistic, large-scale public dataset reflecting this scenario impedes rigorous evaluation and iterative improvement of enterprise-targeted RAG solutions.

EnterpriseRAG-Bench addresses this gap by releasing a synthetic yet structurally representative corpus, a systematic generation pipeline, and a leaderboard for standardized evaluation. The benchmark is positioned to become foundational for both industry and academic research on enterprise-grade RAG.

Corpus Construction and Design Considerations

The EnterpriseRAG-Bench corpus consists of approximately 500,000 documents spanning nine enterprise data sources: Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence. Document generation is informed by a set of foundational principles meant to induce realistic intra-corpus structure, semantic density, and real-world imperfections:

• Cross-document coherence: Documents share initiatives, employees, and projects, emulating real organizational entanglement.
• Realistic source and topic distributions: Slack and Gmail far outnumber curated wikis, reflecting actual enterprise data skews.
• Systematic and plausible noise: Misfiling, near-duplicates, and deliberate information conflicts are synthesized to introduce retrieval challenges often ignored by existing benchmarks.
• Usage of internal jargon and codenames: The corpus embeds organization-specific lexical patterns, necessitating non-trivial semantic matching and increasing retrieval difficulty.
• Generalizable generation pipeline: The methodology allows adaptation to new domains by varying organizational scaffolding, document type mix, and generation scale.

The document set is generated at two fidelity levels: a high-fidelity core for clusters that demand full cross-document coherence and a high-volume stream using topic scaffolding to efficiently scale while avoiding excessive duplication.

Figure 1: The generation pipeline scaffolds and grounds documents on human-in-the-loop artifacts to ensure coherence, enabling cost-efficient scaling with preserved organizational structure.

Query Design and Evaluation Categories

The benchmark defines 500 questions distributed across ten categories, each probing distinct aspects of retrieval, synthesis, reasoning, and robustness:

• Basic: Single document retrieval with moderate keyword overlap.
• Semantic: Roundabout phrasing requires semantic similarity recognition.
• Intra-Document Reasoning: Linking facts from distant sections within one document.
• Project Related: Multi-document extraction within an initiative.
• Constrained: Disambiguation using qualifiers across similar documents.
• Conflicting Info: Reconciliation of contradicting data sources.
• Completeness: Exhaustive retrieval of all relevant documents.
• Miscellaneous: Retrieval from peripheral or informally organized files.
• High Level: Corpus-wide synthesis.
• Info Not Found: Robustness against hallucination when the answer does not exist.

This diverse question suite systematically targets both obvious and subtle RAG failure modes encountered in enterprise deployment.

Structural and Semantic Corpus Analysis

Through t-SNE visualization and quantitative embedding-based similarity metrics, the authors demonstrate that EnterpriseRAG-Bench exhibits local density and topic clustering characteristics closely resembling a real enterprise corpus, and distinct from open-domain benchmarks like BrowseComp-Plus.

Figure 2: t-SNE projections reveal clustered and interconnected semantic structure in both the Onyx production corpus and the synthetic EnterpriseRAG-Bench, contrasting with the diffuse structure of open-domain corpora.

The local average cosine similarity (nearest neighbors) is significantly higher in the enterprise datasets ($0.83$) than in open-Web corpora ($0.69$), corresponding to higher distractor density and consequently greater retrieval challenge—an effect absent from public benchmarks.

Evaluation Harness and Metrics

Ground-truth determination at enterprise scale is inherently revisable due to corpus scale and redundancy. The scoring pipeline features:

• Correction-aware evaluation: Retrieval candidate sets are pooled and gold standards revised via LLM-based majority consensus, with changes propagated to answer/fact extraction.
• Multi-dimensional scoring: Metrics include binary correctness, completeness (fact-level), Recall@10 on document retrieval, and count of invalid extra documents.
• Leaderboard orchestration: Systems are ranked by completeness conditional on correctness, reflecting real enterprise use-case priorities.
• Head-to-head comparative mode: Per-question judgments compare competing systems, supporting nuanced error analysis beyond aggregate scores.

Baseline Results and Empirical Insights

Three retrieval paradigms were benchmarked—BM25, dense vector search with OpenAI’s text-embedding-3-large, and an agentic bash file-system retriever:

Metric	BM25	Vector	Bash Agent
Correctness (%)	68.8	51.4	60.6
Completeness (%)	56.0	42.9	61.1
Document Recall (%)	68.4	46.0	55.8
Invalid Extra Docs	9.0	9.3	2.0

BM25 outperformed dense vector search even in semantic match categories, contradicting the expectation that embedding retrieval would dominate on low-keyword-overlap queries. This suggests that existing embedding models, trained on public data, inadequately capture the lexical, acronymic, and conversational idiosyncrasies of proprietary data.

Agentic retrieval (Bash Agent) achieved the highest completeness, especially on categories requiring iterative exploration (completeness, miscellaneous), although at a significantly higher computational cost. These results highlight that current vector search paradigms may inadequately address the retrieval landscape in enterprise settings, and that efficient RAG on proprietary corpora remains a challenging, open problem.

Scaling Effects in Retrieval Difficulty

Retrieval performance was broadly anti-correlated with corpus size: as corpus size increased from $5$k to $500$k, top-10 nearest neighbor similarity rose, and Recall@10 dropped for both BM25 and vector retrieval.

Figure 3: Increasing corpus size raises nearest-neighbor embedding similarity and depresses Recall@10, reflecting increasing distractor prevalence and retrieval difficulty.

This scaling behavior underscores the importance of realism in both corpus scale and local semantic density for evaluating RAG systems fit for production deployment in enterprises.

Limitations and Future Directions

Key limitations stem from the synthetic nature of the data:

• Per-document realism is bounded by LLM generation capabilities: synthetic artifacts, e.g., uniformity, lack of tangential noise, and over-focused conversations, are present.
• All documents are exported as flat JSON, suppressing complex nesting and rich formatting.
• The corpus simulates a technology company; translation to highly divergent organizational or vertical structures (e.g., regulated or manufacturing sectors) may require further adaptation.

The authors propose extensions targeting multi-modal data, deep multi-hop reasoning, high-volume aggregation, recency-aware, and people-centric queries, as well as continued refinement of evaluation harnesses focused on corrigibility and reproducibility of gold labels as more systems and retrieval paradigms are benchmarked.

Conclusion

EnterpriseRAG-Bench constitutes a substantive advance in benchmark design for enterprise-specific RAG, providing a representative, large-scale synthetic corpus with detailed evaluation resources and a robust correction-aware leaderboard. By exposing genuine pain points in enterprise information retrieval—especially the surprising superiority of BM25 over current dense vector retrieval on internal corpora—the benchmark enables rigorous, domain-aligned progress in retrieval system development. Its public availability and extensible design are likely to foster systematic innovation, improved transparency, and reproducible research in RAG for enterprise deployments.