RAGTrace: Interactive RAG Analysis

Updated 8 July 2026

RAGTrace is an interactive system that provides a holistic evaluation of RAG workflows by analyzing the interplay between retrieval and generation.
It integrates high-level performance metrics with fine-grained visual diagnostics to pinpoint failures in retrieval relevance and generation fidelity.
The tool supports iterative refinement by allowing users to adjust retrieval parameters and trace answer provenance for improved system reliability.

RAGTrace is an interactive evaluation system for analyzing retrieval-generation dynamics in Retrieval-Augmented Generation (RAG) workflows. It is introduced to address a specific opacity problem in RAG: while retrieval accuracy and response quality have received substantial attention, the internal knowledge integration and retrieval-generation interactions in RAG workflows remain largely opaque. Informed by a comprehensive literature review and expert interviews, RAGTrace supports a multi-level analysis approach that ranges from high-level performance evaluation to fine-grained examination of retrieval relevance, generation fidelity, and cross-component interactions. Its workflow allows users to build, evaluate, and iterate on retrieval processes tailored to specific domains of interest, and its effectiveness is demonstrated through case studies and expert evaluations on real-world RAG applications (Cheng et al., 8 Aug 2025).

1. Problem setting and conceptual scope

RAG systems combine a retriever with a generator, but conventional evaluation practices often assess retrieval or generation in isolation. RAGTrace is defined against that limitation: it holistically analyzes the relationship and interplay between retrieval and generation modules, allowing users to trace the provenance of generated content and diagnose whether failures arise from retrieval, generation, prompt sensitivity, or cross-component interaction effects (Cheng et al., 8 Aug 2025).

The system’s central premise is that end-to-end quality scores alone are insufficient for debugging knowledge-grounded systems. Accordingly, RAGTrace provides both high-level performance evaluation and fine-grained assessment of RAG workflows, enabling users to move beyond passive receipt of model outputs to active inspection, diagnosis, and optimization of retrieval-generation interactions. This suggests a complementary relation to benchmark-centric evaluation frameworks such as TRACe in RAGBench, which formalizes four core metrics—Relevance, Utilization, Completeness, and Adherence—for explainable and actionable RAG evaluation (Friel et al., 2024).

A common misconception is to treat RAGTrace as a benchmark or a single metric suite. The paper instead presents it as an interactive system with linked visual analytics, diagnostic metrics, provenance tracing, and iterative optimization support. Its stated emphasis is integrated exploration of retrieval-generation relationships rather than isolated scoring of either component (Cheng et al., 8 Aug 2025).

2. System architecture and workflow

RAGTrace consists of three main interactive components. The first is Visual Exploration for Diagnosis, which includes a heatmap that visualizes chunk coverage and density in semantic embedding space projected via openTSNE, a force-directed graph that clusters and positions questions by error pattern and performance, and a search-and-details pane for targeted exploration. The second is Retrieval Performance and Evidence Traceability, which includes a chunk-relink graph for comparing retrieved evidence across retrieval strategies and interactive evidence chains that link answer components to their provenance. The third is Comparative Optimization, which includes radar charts for before/after comparison and a sampling panel for interactive adjustment of retrieval parameters such as diversity and number of chunks (Cheng et al., 8 Aug 2025).

At the workflow level, the system supports build-evaluate-iterate analysis. Users can compare original, optimized, and gold or reference retrieval strategies; inspect how retrieved chunks are distributed, overlapped, and reused; and drill down from system-level patterns to instance-level failures. Linked views and synchronized highlighting enable movement between global workflow failures and local chunk- or answer-level evidence relationships (Cheng et al., 8 Aug 2025).

RAGTrace also incorporates a concrete data-processing stack. It uses OpenAI’s text-embedding-3-large for semantic chunk and question embeddings, openTSNE for dimensionality reduction and spatial visualization, and named entity extraction and alignment using NLTK together with LLM-driven synonym and antonym expansion. The implementation summary further notes Llama3-70B-4bit for local generation and OpenAI or DeepSeek services for embeddings and entity augmentation (Cheng et al., 8 Aug 2025).

3. Analytical levels, metrics, and visual diagnostics

RAGTrace organizes analysis into multiple levels. At the highest level, it uses composite performance metrics such as BLEU, ROUGE, and GPTScore for end-to-end assessment of output quality. These are visualized in summary dashboards and radar charts to support comparison across RAG configurations (Cheng et al., 8 Aug 2025).

At a finer level, the system introduces granular diagnostic metrics for distinct failure modes. For retrieval relevance, it uses Retrieval Failure Value $(\mathcal{R}_{\text{fail}})$ , which quantifies both the hit rate on key evidence and the entropy of retrieved results, and Prompt Fragility Value, which measures sensitivity to prompt variations by evaluating divergence in retrieved chunks across paraphrases. For generation fidelity, it uses Generation Anomaly Value $(\mathcal{A}_{\text{gen}})$ , combining model output confidence with the ratio of hallucinated or erroneous citations in the generated output. For factual drift and unsupported claims, it uses Standard Hallucination Value $(R_{\text{hall}})$ , which combines GPTCheck and FactScore (Cheng et al., 8 Aug 2025).

The paper gives explicit forms for three of these quantities:

$\mathcal{R}_{\text{fail}} = \alpha \frac{\sum_{c_k \in C_{\text{gold}}} \mathcal{T}_\theta(\text{sim}(c_k, C_{\text{ret}}))}{|C_{\text{gold}}|} + \beta \frac{1}{n} \sum_{i=1}^{n} \text{Entropy}(\text{sim}(c_i, C_{\text{ret}}))$

$\mathcal{A}_{\text{gen}} = \alpha \cdot (\text{Mean Confidence}) + \beta \cdot (\text{Error Chunk Ratio})$

$R_{\text{hall}} = \alpha \cdot \text{GPTCheck}(\text{output}) + \beta \cdot \text{FactScore}(\text{output})$

These metrics are paired with visual analytics rather than used as isolated scalar judgments. Force-directed graphs cluster questions according to retrieval failure, prompt vulnerability, generation anomaly, and standard inconsistency. Heatmaps and chunk-relink graphs display the distribution, overlap, and relevance of retrieved chunks between retrieval strategies. Evidence traceability allows users to inspect how retrieved evidence is used or not used by the generation step, surfacing mismatches and lost knowledge (Cheng et al., 8 Aug 2025).

The metric layer is extensible. The paper states that six key metrics are used—four granular by error type and two composite—and that more can be added by users. This makes the framework adaptable to domain-specific evaluation practices rather than fixed to a closed benchmark protocol (Cheng et al., 8 Aug 2025).

A defining feature of RAGTrace is its notion of traceability. Users can trace any part of the generated answer directly to its supporting or missing retrieved chunks. In practice, this means that attribution errors and hallucinations appear as missing or low-similarity links in the interface, rather than only as degraded end-task scores. This reorients evaluation from answer-only inspection to provenance-aware diagnosis (Cheng et al., 8 Aug 2025).

The system supports failure diagnosis by allowing users to switch between views and determine whether an error stems from poor retrieval, prompt ambiguity, or hallucinated or mis-attributed generation. It also supports optimization through parameter adjustment and visual comparison, including re-ranking changes, prompt adjustment, and knowledge-base expansion, with real-time feedback on performance improvement (Cheng et al., 8 Aug 2025).

This interactive refinement loop differentiates RAGTrace from infrastructure-oriented trace datasets. RAGPulse, for example, introduces an open-source RAG workload trace in which each request records timestamp, input_length, output_length, hash_ids, and session_id, revealing temporal locality and a highly skewed hot document access pattern for serving-system optimization (Wang et al., 17 Nov 2025). RAGTrace addresses a different layer: retrieval-generation relationships, provenance, and failure diagnosis at the evaluation and debugging interface rather than workload replay for systems optimization.

A plausible implication is that RAGTrace can function as an observability layer over domain-specific RAG deployments, especially where developers need to validate retrieval modifications against downstream generation behavior rather than against retrieval metrics alone. That implication is consistent with the paper’s stated support for iterative, domain-tailored retrieval design (Cheng et al., 8 Aug 2025).

5. Empirical validation and case studies

The paper reports a user study with 11 participants, including LLM or RAG researchers and domain experts, who used RAGTrace to analyze questions from the Natural Questions dataset with a Wikipedia-based RAG system. In that study, the PSSUQ usability mean was 6.18/7, and NASA-TLX showed low cognitive and physical demand together with high perceived task success, with a mean of 16.73/20. Participants also reported “substantial improvements” in debugging efficiency, including 80% time improvement for retrieval diagnosis for some users (Cheng et al., 8 Aug 2025).

The qualitative findings emphasize the value of visual provenance, error clustering, and interactive parameterization. Participants highlighted support for both low-level analysis, such as chunk and retrieval ranking inspection, and high-level analysis, such as question type and system configuration effects. Evidence tracing and chunk comparison were identified as critical features for debugging and iterative improvement (Cheng et al., 8 Aug 2025).

The case studies illustrate the intended analytic granularity. One case traced entity fragmentation in “Tom and Jerry” fact queries. Another examined iterative improvements obtained by applying HyDE as a retrieval optimization, with the effects observed through visualization and improved performance metrics. A further case focused on domain-specific optimization for medical queries by augmenting the corpus and refining retrieval, again with improvements confirmed through dashboard analytics (Cheng et al., 8 Aug 2025).

These results do not merely validate interface usability. They support the stronger claim that integrated visual and metric analysis can alter debugging practice by making provenance relationships inspectable and by reducing the time required to isolate retrieval-related errors (Cheng et al., 8 Aug 2025).

6. Position within the trace-oriented RAG literature

The term “trace” is used in several distinct ways across RAG research. RAGTrace belongs specifically to interactive evaluation of retrieval-generation dynamics, and it should not be conflated with benchmark metrics, reasoning-chain construction, workload tracing, or security traceback mechanisms.

System	Traced object	Primary purpose
RAGTrace	Retrieval-generation relationships and evidence provenance	Interactive evaluation and refinement (Cheng et al., 8 Aug 2025)
TRACe in RAGBench	Relevance, Utilization, Completeness, Adherence	Explainable RAG benchmarking (Friel et al., 2024)
TRACE	Knowledge-grounded reasoning chains of triples	Multi-hop QA improvement (Fang et al., 2024)

Other contemporaneous systems extend the “trace” idea in different directions. ARENA enforces a structured response format with <relevance>, <analysis>, and <answer> so that chosen references, reasoning traces, and answers are auditable, and trains that behavior with reinforcement learning and adaptive rewards (Ren et al., 19 May 2025). TaSR-RAG decomposes a query into ordered triple-form sub-queries, performs step-wise evidence selection, and maintains an explicit entity binding table, yielding faithful reasoning traces for multi-hop questions (Sun et al., 10 Mar 2026). CyberRAG logs iterative retrieval, evidence selection, and re-retrieval decisions in an agentic cyber-attack classification pipeline, making orchestration auditable (Blefari et al., 3 Jul 2025).

A separate branch of work uses “trace” in security and forensics. TRACE identifies poisoning attacks by tracing answer-related tokens through token influence attribution and can uncover attacker-specified target answers (Chen et al., 24 Jun 2026). “Needle-in-RAG” introduces RAGCharacter, a two-pass forensic framework that localizes responsible poisoned spans at the character level through prompt-anchored execution traces and counterfactual masking with replay (Cui et al., 3 May 2026). In software security, TraceRAG bridges natural-language behavior queries to method-level Java code and produces human-readable reports that link malicious behaviors to code implementations (Zhang et al., 10 Sep 2025).

This broader landscape clarifies RAGTrace’s niche. It is neither a reasoning-chain constructor nor a poisoning detector nor a workload dataset. It is a unified, interactive, and multi-level analytical environment for understanding and optimizing RAG workflows through visual provenance tracing, diagnostic metrics, and iterative retrieval refinement (Cheng et al., 8 Aug 2025).