RAGalyst: Evaluation for RAG Systems

• RAGalyst is an automated evaluation framework for RAG systems that integrates synthetic QA dataset generation, agentic filtering, and LLM-as-a-Judge evaluation.
• It benchmarks retrieval quality and generation accuracy using metrics like Answer Correctness, Answerability, and Faithfulness, all validated against human annotations.
• The framework offers actionable insights for model selection, prompt optimization, and error analysis across diverse domains such as military, cybersecurity, and engineering.

RAGalyst is an automated, agentic evaluation framework for Retrieval-Augmented Generation (RAG) systems, specifically designed to address the unique challenges of rigorous, human-aligned evaluation in domain-specific and safety-critical contexts. Its primary innovation lies in combining an agentic synthetic data pipeline with LLM-as-a-Judge metrics refined through prompt optimization, enabling assessment of both retrieval and generation components using metrics validated for alignment with human annotations. RAGalyst systematically benchmarks retrieval quality, answer generation, and error modalities across a range of domains, providing empirical foundations for model selection, prompt design, and system configuration to maximize reliability and relevance in real-world deployments (Gao et al., 6 Nov 2025).

1. Agentic Evaluation Pipeline

RAGalyst employs a three-stage process to generate domain-specific benchmarks and evaluate RAG pipelines:

1. Synthetic QA Dataset Generation: Input documents (PDF, markdown, text) are segmented into overlapping chunks (max 800 tokens, 400 token overlap) using LangChain. A "user" LLM agent drafts questions tailored to each chunk, while an "expert" LLM agent generates fact-grounded answers referencing only the chunked context.
2. Agentic Filtering: Each proposed (Question, Answer, Context) (QAC) triplet is evaluated using LLM-as-a-Judge metrics—Answerability, Faithfulness, and Answer Relevance. Only QAC samples meeting tunable metric thresholds are retained; higher thresholds yield higher-quality datasets at the expense of throughput.
3. LLM-as-a-Judge Evaluation: The resulting filtered dataset is used both to optimize and validate metric prompts and to benchmark RAG pipeline performance: for each test sample, retrieved documents and generated answers are scored on Answer Correctness, Answerability, Faithfulness, and Answer Relevance. Retrieval quality is measured with Recall@K and MRR@K (Gao et al., 6 Nov 2025).

2. Metric Formalization and Human Alignment

RAGalyst defines its core LLM-as-a-Judge metrics at both the mathematical and empirical levels:

• Answer Correctness: A continuous score in $[0, 1]$ assessing the semantic agreement between a generated answer $a_{\mathrm{gen}}$ and a reference answer $a_{\mathrm{ref}}$:

$$\mathrm{Correctness}(a_{\rm gen},a_{\rm ref}) = \text{LLM\_score}\bigl(a_{\rm gen},a_{\rm ref}\bigr)\in[0,1].$$

Human judge alignment is evaluated via Spearman's rank correlation $\rho_s$ between LLM scores and ground-truth human labels:

$$\rho_s = 1 - \frac{6\sum_i d_i^2}{n(n^2-1)}, \quad d_i = \mathrm{rank}(\mathrm{Correctness}_i) - \mathrm{rank}(h_i).$$

The standard error of $\rho_s$ follows Bonett & Wright (2000):

$$\mathrm{SE}(\rho_s) = \sqrt{\frac{1 + \tfrac12\,\rho_s^2}{\,n-3\,}}.$$

• Answerability: A binary flag determining if question $q$ is fully answerable using context $c$:

$a_{\mathrm{gen}}$0

Agreement with human annotations is also reported via Spearman's $a_{\mathrm{gen}}$1 on SQuAD 2.0 (Gao et al., 6 Nov 2025).

Empirical optimization achieves high alignment: for instance, Answer Correctness $a_{\mathrm{gen}}$2 (STS-B, $a_{\mathrm{gen}}$3), outperforming cosine similarity and the RAGAS metric. For Answerability, the best achieved $a_{\mathrm{gen}}$4 is $a_{\mathrm{gen}}$5 (SQuAD 2.0).

3. Prompt Optimization and Judge Model Selection

RAGalyst refines Judge prompts using DSPy’s automatic optimization suite:

• Base Prompting: The hand-crafted Answer Correctness prompt instructs scalar scoring (0.0–1.0) with examples.
• Instruction Search: COPRO (coordinate ascent) and MIPROv2 (Bayesian search) automatically search prompt wordings. Both use GPT-4o-mini ($a_{\mathrm{gen}}$6) as the underlying model.
• Few-Shot Demonstration Tuning: The "LabeledFewShot" optimizer injects $a_{\mathrm{gen}}$7 human-labeled QAC examples into the prompt; $a_{\mathrm{gen}}$8 yields optimal Answer Correctness correlation.
• Final Prompt Selection: For Answer Correctness, MIPROv2-optimized instructions with 8 few-shot instances are used. Neither COPRO nor MIPROv2 improved Answerability, so its prompt remains a simple 0/1 rubric with hand-selected examples.

Judge model ablation across Gemini, Qwen3, and GPT-4.1 families found that GPT-4o-mini offers the best tradeoff in correlation, speed, and cost, and is standardized for both optimization and evaluation stages (Gao et al., 6 Nov 2025).

4. Domain-Diverse Evaluation and Empirical Findings

RAGalyst has been extensively benchmarked across three knowledge domains: military operations (Army), cybersecurity, and bridge engineering (Engineering).

• Synthetic QA Dataset Quality: Compared to the RAGAS baseline, RAGalyst-generated datasets score substantially higher in Answerability and Faithfulness (e.g., in Cybersecurity, Answerability 0.974 vs. 0.306; Faithfulness 0.991 vs. 0.797).
• Retrieval Model Performance: Embedding models—text-embedding-3, Gemini-emb, BGE-M3, Qwen3 (0.6B/4B/8B), nomic-embed—were evaluated. Qwen3-4B achieves the highest Recall@10 in most domains. Domain-dependence is observed, with Qwen3 models maintaining performance in cybersecurity where others degrade.
• LLM Generation Benchmarks: With retrieval fixed to Qwen3-8B, several LLMs (Gemma3-27B, Qwen3-30B, Gemini-2.5 variants, GPT-4o-mini, GPT-4.1 variants) were compared:
  • Gemini-2.5-flash attains highest Answer Correctness and Faithfulness in all domains (Correctness: 0.86/0.85/0.88).
  • GPT-4.1-nano achieves the highest Answer Relevancy (0.93–0.95).
  • No single LLM dominates across all metrics or domains; closed- vs. open-source performance is inconsistent; model size is not predictive.
• Chunk Retrieval Ablation: Varying the number of retrieved chunks (1–10), Faithfulness declines slightly at higher chunk counts, Relevancy increases, and Correctness peaks at 3–5 chunks.
• Analysis of Low Correctness Cases: Failure modes, annotated using GPT-5, are primarily due to RAG-side Over-Specificity (71%), followed by extraction failures and LLM-side context inconsistency or factual fabrication. Over-Specificity is a result of synthetic QA generator constraint to a single chunk, versus the RAG system answering from multi-chunk evidence (Gao et al., 6 Nov 2025).

5. Design Principles and Modality Extensions

RAGalyst provides empirically grounded recommendations for building and tuning RAG systems:

• Embedding Model Selection: Always benchmark Recall@K and MRR@K on the target domain; high general MTEB rank does not guarantee domain performance. Qwen3-4B and Qwen3-8B are strong choices; for constrained environments, smaller models may suffice.
• LLM Generation: No universal optimal model; selection is metric-, domain-, and budget-dependent.
• Retrieval Depth: Optimal number of chunks is usually 3–5 for single-hop QA. More chunks may improve coverage but degrade answer focus.
• Prompt Optimization: For scalar metrics (e.g., Answer Correctness), combine prompt-search and few-shot tuning. For categorical metrics (e.g., Answerability), hand-crafted rubrics are robust.
• Filtering Thresholds: For data generation, use strict thresholds when possible (e.g., Answerability ≥0.95) to maximize quality; lower thresholds expedite generation at some cost to data fidelity (Gao et al., 6 Nov 2025).

Integration with multimodal benchmarks such as FATHOMS-RAG (Hildebrand et al., 10 Oct 2025) extends RAGalyst’s applicability. The FATHOMS-RAG dataset comprises 93 questions spanning text, tables, images, within- and cross-document multimodality, each requiring concrete grounding in the source. Phrase-level recall is used for correctness scoring:

$a_{\mathrm{gen}}$9

Hallucinations are detected using a nearest-neighbor SentenceTransformer classifier combined with phrase-level recall (Hildebrand et al., 10 Oct 2025). This suggests direct extensibility of RAGalyst to text+table+image multi-hop QA and hallucination detection by incorporating FATHOMS-RAG metrics and datasets.

6. Strategic Value and Scope of Application

RAGalyst is positioned as both a benchmarking tool and a decision-support platform for RAG deployments in specialized domains. Its modularity enables practitioners to rapidly prototype, tune, and validate retrieval and generation pipelines against synthetic or real data, quantify domain-specific trade-offs, and identify error modes. The framework’s design allows seamless scaling to new modalities (e.g., tables, images, scientific metadata) and integration with multimodal benchmarks, facilitating comprehensive stress-testing of document granularity, evidence integration, and hallucination detection (Gao et al., 6 Nov 2025, Hildebrand et al., 10 Oct 2025).

RAGalyst's empirical findings underscore the context-dependence of RAG components and configurations, advocating for systematic, domain-sensitive evaluation rather than reliance on leaderboard or general-purpose rankings. Its open-source availability enables rapid adoption and iterative development for practitioners seeking robust evaluation tailored to complex, real-world information environments.