RAG-Eval: Modular Evaluation of RAG Systems

• RAG-Eval is a modular framework that decomposes evaluation across indexing, retrieval, and generation components for nuanced diagnostics.
• It combines automated LLM-based scoring with human and synthetic data tests to benchmark system performance effectively.
• The approach integrates diverse metrics and iterative refinement strategies to address domain-specific challenges and ensure robust evaluations.

RAG evaluation, often abbreviated as RAG-Eval, denotes the systematic assessment of Retrieval-Augmented Generation systems across indexing/database, retrieval, and generation components. The literature treats this assessment as modular and multi-dimensional, combining component-wise evaluation, dataset-driven benchmarking, automated LLM-based evaluation, human evaluation, and hybrid protocols. A recurring theme is that there is currently no generally accepted methodology for RAG evaluation, particularly in enterprise and domain-specific settings where gold standards are unavailable or costly to obtain (Brehme et al., 28 Apr 2025, Simon et al., 2024, Rackauckas et al., 2024).

1. Evaluation object and methodological decomposition

A central premise of the recent literature is that RAG systems should not be evaluated as monolithic pipelines alone. Because RAG involves multiple components—indexing/database, retriever, and generator—evaluation is typically organized component-wise, with customized metrics relevant to each function. For retrievers, commonly used metrics include Recall, Precision, F1-score, Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Discounted Cumulative Gain (NDCG), and Hit Rate. For generators, the literature uses Exact Match, accuracy, F1-like metrics, semantic similarity measures such as SAS, SBERT, and BERTScore, n-gram metrics such as BLEU and ROUGE, and LLM-as-a-Judge scoring for faithfulness, correctness, relevance, hallucination, and completeness (Brehme et al., 28 Apr 2025).

One reusable methodological blueprint structures evaluation around independent variables, dependent variables, baseline selection, benchmark construction, holdout testing, and iterative refinement. Independent variables include context resources and RAG design choices such as embedding models, search and retrieval strategies, filters, ranking algorithms, and prompt templates. Dependent variables should be task-specific rather than limited to retrieval quality alone. The same blueprint emphasizes that baselines are crucial, that prompt improvements applied to RAG variants should also be applied to the baseline LLM for a fair comparison, and that final reported results should come from evaluation on unseen holdout data after qualitative failure analysis has guided refinements (Simon et al., 2024).

This decomposition is reinforced by work on retrieval-specific evaluation. The eRAG approach evaluates retrieval by feeding each document in the retrieval list individually to the LLM, comparing the output against downstream ground truth, and treating downstream performance for each document as its relevance label. This redefines document utility in terms of actual task performance rather than static relevance labels, and the reported correlation with downstream RAG performance exceeds baseline methods while reducing runtime and memory relative to end-to-end evaluation (Salemi et al., 2024).

2. Datasets, benchmarks, and synthetic evaluation corpora

Datasets are foundational in RAG evaluation because they determine which kinds of knowledge, reasoning, and failure modes are measured. The literature distinguishes public QA datasets such as HotPotQA, Natural Questions, MS MARCO, and similar resources from domain-specific or custom datasets built for proprietary or niche domains. It also classifies benchmarks into short-answer, long-form, multi-hop, error-type or adversarial, and misleading-context or unanswerable settings. A consistent caution is that public datasets may be inexpensive and convenient, but they can be non-specific and may even permit pretraining leakage (Brehme et al., 28 Apr 2025).

A more granular view comes from a dataset taxonomy that labels context–query pairs as Fact, Summary, Reasoning, or Unanswerable. In that taxonomy, public datasets are reported as highly unbalanced, with over-representation of fact questions and scarcity of reasoning and summary types. The same work shows that retrieval performance varies substantially by label and that the difference in best recall between query types within the same dataset can be as large as 42%. This suggests that aggregate retrieval scores can obscure use-case-specific weaknesses, especially when the query distribution in deployment differs from the query distribution in the benchmark (Lima et al., 2024).

Because many enterprise domains lack annotated QA pairs, synthetic data generation has become a major subfield of RAG evaluation. RAGElo generates synthetic user queries in bulk by prompting LLMs such as GPT-4 Turbo and Claude Opus/Sonnet/Haiku with large passages sampled from company documents and with authentic user queries as few-shot examples; some domain experts also supply exemplar queries. RAGalyst similarly uses an agentic synthetic QA pipeline in which one LLM generates a question, another produces a context-grounded answer, and an agentic filtering step retains only samples that meet thresholds on Answerability, Faithfulness, and Answer Relevance. The reported throughput trade-off is explicit: 0.88 spm single-threaded for the filtered pipeline versus 14.6 spm for RAGAS-style generation (Rackauckas et al., 2024, Gao et al., 6 Nov 2025).

Synthetic evaluation can also be used to stress-test systems through deliberately difficult scenarios. RAGProbe defines an evaluation scenario schema with six constructs: document sampling strategy, chunking strategy, chunk sampling strategy, scenario-specific prompts, prompting strategy, and acceptable evaluation metrics. It then instantiates six QA scenarios, including numeric questions, date/time questions, multiple-choice questions, combined questions from a single document, combined questions spanning multiple documents, and questions with no answer in the corpus. Across five open-source RAG pipelines, the highest failure rates were reported for combined questions spanning multiple documents (91%), combined questions from a single document (78%), and no-answer scenarios (65%). Relative to RAGAS, the approach exposed 51% more failures per dataset on average (Sivasothy et al., 2024).

3. Metrics and judging protocols

RAG evaluation uses both retrieval-oriented and answer-oriented metrics, and recent work increasingly relies on LLMs as evaluators. In RAGElo, a judge LLM rates retrieved documents pointwise for relevance and rates answers using four dimensions—Relevance, Accuracy, Completeness, and Precision—on a 0–2 scale. The same framework also uses pairwise answer comparisons in an Elo-inspired tournament, enabling head-to-head ranking of RAG agents without gold-standard answers. For retrieval, it reports MRR@5, defined as

$$\text{MRR@5} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\operatorname{rank}_i},$$

where $\operatorname{rank}_i$ is the position of the first relevant document for query $i$ if it appears in the top 5 (Rackauckas et al., 2024).

A complementary line of work refines metric transparency. In the telecom domain, a modified version of RAGAS exposes intermediate outputs for faithfulness, context relevance, answer relevance, answer correctness, answer similarity, and factual correctness. There, Faithfulness is computed as

$\text{Faithfulness} = \frac{|V|}{|S|},$

where $|V|$ is the count of supported statements and $|S|$ is the total number of statements. The same study reports that Faithfulness and Factual Correctness aligned best with SME evaluations, whereas Answer Similarity, Answer Relevance, and Context Relevance were less reliable because cosine similarity and context-length effects were hard to interpret in the telecom setting (Roychowdhury et al., 2024).

Zero-shot, end-to-end judge frameworks further simplify scoring. CCRS defines five LLM-as-a-Judge metrics: Contextual Coherence, Question Relevance, Information Density, Answer Correctness, and Information Recall. Its Answer Correctness metric combines exact match and LLM judgment as

$$AC(r, g, C) = \lambda \times EM(r, g) + (1 - \lambda) \times \frac{LLM_\text{Judge}(r, g, C, \text{prompt}_\text{AC})}{100},$$

with $\lambda = 0.7$. On BioASQ, CCRS is reported as comparable or superior in discriminative power to RAGChecker for key aspects such as recall and faithfulness, while being 5x faster in the authors’ setup (Muhamed, 25 Jun 2025).

Reference-free hallucination detection constitutes another important metric family. A benchmark across six RAG applications evaluates LLM-as-a-Judge, Prometheus, Lynx, HHEM, and TLM using AUROC. In that benchmark, TLM is consistently the top performer on FinQA, ELI5, FinanceBench, PubmedQA, CovidQA, and DROP, while LLM-as-a-Judge is the second best overall. The benchmark is notable because all compared methods are reference-free and operate on the query, retrieved context, and generated response alone (Sardana, 27 Mar 2025).

4. Frameworks, platforms, and reproducible tooling

The current ecosystem includes ranking frameworks, introspection platforms, playground-style benchmarking environments, and local-first evaluation libraries. Together they reflect a shift from single metrics toward diagnostic environments that expose failures at aggregate, instance, annotator, and component levels (Fadnis et al., 2024, Papadimitriou et al., 2024, Strich et al., 31 Oct 2025).

Framework	Primary target	Distinctive element
RAGElo	Relative ranking of RAG agents	Automated Elo-based competition
eRAG	Retrieval quality	Per-document downstream labeling
InspectorRAGet	Result analysis	Aggregate, instance, and annotator views
RAG Playground	Retrieval and prompting strategies	Multi-metric pass-rate evaluation
EncouRAGe	Local, reproducible benchmarking	20+ metrics and modular components
CCRS	End-to-end answer quality	Zero-shot LLM-as-a-Judge suite

InspectorRAGet is explicitly oriented toward the analysis step of evaluation. It supports aggregate performance, instance-level analysis, mixed human and algorithmic metrics, annotator qualification, and dataset characterization. Its interface includes radar charts, score distributions, pairwise model comparison with Fisher’s randomization test, metric correlation views using Spearman coefficients, and annotator analyses based on Cohen’s Kappa. In the examples reported, it surfaced model-ranking discrepancies between algorithmic metrics and human preferences, as well as annotator disagreement patterns (Fadnis et al., 2024).

RAG Playground and EncouRAGe focus more directly on reproducible experimentation. RAG Playground compares naive vector search, reranking, and hybrid vector-keyword search together with ReAct agents and structured self-evaluation prompts; the best reported configuration reaches a 72.7% pass rate on its multi-metric framework. EncouRAGe provides Type Manifest, RAG Factory, Inference, Vector Store, and Metrics modules, evaluates more than 25k QA pairs and over 51k documents, and reports that Hybrid BM25 consistently achieves the best results across all four datasets, while reranking yields only marginal performance improvements together with higher response latency (Papadimitriou et al., 2024, Strich et al., 31 Oct 2025).

5. Enterprise, domain-specific, and high-stakes evaluation

Enterprise and safety-critical deployments have driven evaluation protocols beyond single-turn benchmark QA. In Infineon’s product QA setting, RAGElo was designed for company internal tasks lacking labeled benchmarks. LLM-as-a-judge ratings on a random sample of synthetic queries showed a moderate, positive correlation with domain expert scoring, with Kendall’s $\tau \approx 0.56$ and Spearman’s $\rho \approx 0.59$, both with $\operatorname{rank}_i$0. The same study found that RAG-Fusion outperformed baseline RAG in Elo score and significantly outperformed it in completeness, but underperformed in precision; BM25 keyword search also outperformed dense vector search in that technical domain (Rackauckas et al., 2024).

For enterprise multi-turn workflows, a case-aware LLM-as-a-Judge framework introduces eight operationally grounded metrics: Hallucination (Grounding Fidelity), Retrieval Correctness, Context Sufficiency, Answer Helpfulness, Answer Type Fit, Identifier Integrity, Case Issue Identification, and Resolution Alignment. Each metric is scored in $\operatorname{rank}_i$1, severity bands map scores to Severe, Moderate, Minor, and Excellent, and the final score is a weighted sum

$\operatorname{rank}_i$2

The framework uses deterministic prompting at temperature $\operatorname{rank}_i$3 and strict JSON outputs, and the reported human validation reaches approximately 85–90% agreement on critical operational metrics. A central claim of this line of work is that generic proxy metrics produce ambiguous signals in enterprise workflows (Chhabra et al., 23 Feb 2026).

High-risk evaluation protocols also combine human and automated annotations with statistical guarantees. In due diligence for an investment fund, one protocol generates 20 responses per question, applies identical binary prompts to human annotators and a GPT-4o-based LLM-Judge, and measures hallucinations, off-topic responses, failed citations, abstentions, language issues, and partial answers. It then uses Prediction Powered Inference with the estimator

$\operatorname{rank}_i$4

together with theoretically guaranteed confidence intervals. The paper emphasizes that the benefit of PPI is conditional on strong LLM-Judge–human agreement; when agreement is only moderate, the reduction in uncertainty is modest (Martinon et al., 29 Jul 2025).

Clinical evaluation adds yet another dimension: reasoning fidelity. CARE-RAG studies Written Exposure Therapy guidelines using 33 clinician-vetted questions under three context conditions—right context, right context plus noise, and wrong context—for 99 test cases in total. It measures Accuracy, Consistency, and Fidelity, where Fidelity is the degree to which reasoning steps are supported by retrieved evidence. The reported result is explicit: correct retrieval is not sufficient, and no model reaches perfect inference across all reasoning levels (Potluri et al., 20 Nov 2025).

User-centered evaluation remains relevant in deployed systems. A study of five domain-specific RAG applications in governance, cybersecurity, agriculture, industrial research, and medical diagnostics uses a web-based evaluation with 100 participants and six dimensions: Ease of Use, Relevance, Transparency, Responsiveness, Accuracy, and Likelihood of Recommendation. The authors also document twelve lessons learned spanning technical, operational, and ethical challenges, including OCR quality, chunking trade-offs, vector database scalability, event logging, data deduplication, source attribution, and dataset bias (Hasan et al., 25 Jun 2025).

6. Limitations, controversies, and methodological directions

A consistent conclusion across the literature is that automation is feasible but not self-validating. The survey literature argues for balancing automated LLM-based evaluation and human judgment, especially in nuanced or domain-specific scenarios, and recommends documenting prompt templates, scoring rubrics, model versions, and dataset limitations. It also recommends modular evaluation, domain-tailored datasets, adversarial or misleading-context tests, and continuous reevaluation as models and business needs change (Brehme et al., 28 Apr 2025).

Recent empirical work sharpens that caution. One applied study compares metrics from Ragas, DeepEval, RAGChecker, and Opik against human scores and recall on a French business QA dataset. Its key warning is stated directly: Correlation ≠ Validity as Intended. Because the experiment uses only one RAG system, the authors note that high metric–human correlation may reflect question easiness or extractability rather than genuine measurement of answer quality. The proposed remedies are to evaluate at least two RAG systems, use preference tests on pairs of outputs, and complement system-level correlation with input-normalized correlation across systems (Brabant, 8 Jul 2026).

The same skepticism appears in framework-specific papers. RAGElo reports that its judge positively aligns with human annotators, though due caution is still required; its scores are best interpreted as relative rankings rather than substitutes for human labels when absolute judgments are needed. RAGalyst likewise concludes that no single embedding model, LLM, or hyperparameter configuration proves universally optimal, and that performance is highly context-dependent across military operations, cybersecurity, and bridge engineering (Rackauckas et al., 2024, Gao et al., 6 Nov 2025).

Taken together, these findings position RAG-Eval not as a single metric or library, but as a research area centered on methodological rigor, domain alignment, and diagnostic specificity. The field increasingly favors combinations of retrieval metrics, answer metrics, LLM-as-a-Judge scoring, human validation, synthetic data generation, and statistically disciplined comparison, while treating generic proxy scores and single-system correlations with caution.