RAGAPHENE: Chat-Based RAG Annotation
- RAGAPHENE is a chat-based annotation platform designed to evaluate multi-turn Retrieval-Augmented Generation systems with live human interventions.
- It integrates dynamic retrieval with real-time editing, enabling annotators to correct both evidence selection and generated responses.
- The platform supports modular retrievers and generators and has been validated at scale for constructing high-quality conversational benchmarks.
RAGAPHENE is a chat-based annotation platform for Retrieval-Augmented Generation (RAG) systems, designed primarily for benchmarking and evaluating LLMs in multi-turn conversational settings where factual correctness is important. Its name expands to “A RAG Annotation Platform with Human ENhancements and Edits,” reflecting a workflow in which annotators do not merely log interactions or assign static labels, but actively converse with a live RAG system and then repair, enrich, and validate both retrieved evidence and generated responses. The platform is intended to produce realistic, high-quality conversational benchmarks for domain-specific and enterprise RAG deployments, especially those built on custom retrievers, generators, prompts, and proprietary corpora (Fadnis et al., 22 Aug 2025).
1. Conceptual basis and problem setting
RAGAPHENE is motivated by the claim that realistic conversational benchmarking for RAG remains underserved by existing tools. The underlying problem is not only hallucination in LLMs, but the compounded failure modes of retrieval-backed systems: retrieval can miss the right evidence or return irrelevant evidence, and generation can misuse correct evidence and produce unsupported claims. In multi-turn dialogue, these problems are amplified because later turns depend on earlier answers, include clarifications and follow-ups, and may shift the information need across documents or subtopics. An early incorrect or insufficient answer can therefore distort later turns as well (Fadnis et al., 22 Aug 2025).
The platform is positioned against two limitations in prior evaluation practice. First, many annotation interfaces support only shallow conversational annotation, such as thumbs up/down, metadata entry, or simple tagging, without dynamic retrieval and generation in the loop. Second, some grounded-dialogue interfaces provide live generation but do not include an actual retrieval component or mechanisms for correcting retrieved evidence. RAGAPHENE addresses this by coupling a live RAG stack with human intervention. The annotator can preserve the realism of interaction with a deployed-style system while still correcting retriever and generator failures before the conversation is finalized (Fadnis et al., 22 Aug 2025).
This design places “human enhancements and edits” at the center of benchmark construction. Rather than treating model outputs as fixed artifacts to be rated post hoc, the platform treats human repair as part of creating a stronger benchmark. A plausible implication is that the resulting data can supervise both retrieval behavior and response quality more directly than static prompt-response collections.
2. System architecture and implementation
Architecturally, RAGAPHENE follows a standard RAG pipeline wrapped in an annotation-centric interface. A corpus is selected together with a retriever and a generator; the annotator then chats with the agent; the retriever fetches passages from the corpus; and those passages are supplied to the generator to produce a response. Conversations can later be exported in structured JSON for review, experimentation, and offline analysis (Fadnis et al., 22 Aug 2025).
The system is deliberately modular. It supports integration with different retrieval systems and generators, allowing adaptation to domain-specific use cases. In the default setup described by the authors, the retriever is ELSERv1 via ElasticSearch 8.10 over the MTRAG corpora, and the generator is Mixtral 8x7B Instruct. Supported component examples include BM25, ELSER, Llama 3, and GPT-4. The appendix adds implementation details: the platform is a React web application built with NextJS 14, uses the Carbon Design System for the UI, requires Python 3.10+ with minimal dependencies for experiments, and has built-in connectivity to ElasticSearch, MongoDB Atlas, IBM Cloudant, WatsonX.AI, and OpenAI via Node SDKs, with planned support for ChromaDB, Anthropic Claude, and vLLM. It is described as lightweight enough to run on a VM or even a laptop with 2 CPUs and 8GB RAM, with heavier resource usage concentrated in experiment runs (Fadnis et al., 22 Aug 2025).
A useful summary of the main operational modes is given below.
| Mode | Primary function | Notable constraint |
|---|---|---|
| Create | Build live multi-turn RAG conversations | Retrieval and generation are active |
| Review | Quality-control existing conversations | Reviewers may not change user questions or search for new passages |
| Experiment | Run lightweight evaluation | Datasets are restricted to at most 100 tasks |
This architecture is significant because it combines live-system realism with post hoc correction, rather than separating annotation from system behavior. The platform therefore captures retriever configuration, generator configuration, prompts, and turn-by-turn evidence within the same workflow.
3. Conversation creation and human intervention
Create mode is the core annotation workflow. The interface places the chat area on the left and retriever/generator configuration settings on the right. Annotators can choose the document collection, set how many passages to retrieve, adjust query formulation and result rendering, choose the model, edit prompts, and tune decoding or token limits. The turn-level procedure begins with the annotator writing a user question without seeing documents; the paper explicitly states that the user is expected to have some domain knowledge but to ask naturally rather than reverse-engineering from known passages. The system then retrieves passages and generates an answer (Fadnis et al., 22 Aug 2025).
After generation, the annotator can intervene in four main ways. First, retrieved passages can be edited. Annotators may mark passages as irrelevant or discard them, and they may add relevant passages that the retriever missed. To support this, the platform includes a side search interface in which annotators try alternate formulations of the question and browse the corpus for better evidence. After the passage set is changed, the response can be regenerated on the updated evidence. Second, the agent response itself can be edited if it remains wrong, incomplete, poorly phrased, or unfaithful even after evidence correction; the interface displays a diff between the original and edited response. Third, the system highlights lexical overlap between the response and retrieved passages as a lightweight faithfulness aid. Fourth, annotators can add turn-level enrichments such as question type, answerability, and multi-turn category; the paper gives examples including factoid or opinion, answerable or unanswerable, and clarification or follow-up (Fadnis et al., 22 Aug 2025).
The platform also nudges annotators toward completion and consistency. Tips appear during creation to remind them to mark relevant passages or add enrichments when these are omitted, and export-time checkboxes and conversation statistics are presented to encourage higher-quality submissions. This suggests that the platform treats realism and correction as complementary rather than opposed: user turns are intended to be natural, but the final benchmark instance is still curated.
A central methodological feature is that the conversation is created in interaction with a live RAG assistant and only afterward refined toward what a grounded, faithful system ought to have produced. The resulting passage sets are therefore not merely retrieved contexts but human-validated relevance judgments attached to conversational turns.
4. Review, validation, and experiment workflows
Review mode is the main quality-assurance mechanism. The authors emphasize that conversation creation is sophisticated and error-prone, and that evaluation benchmarks must therefore undergo explicit validation. In Review mode, a reviewer receives a batch of existing conversations; retrieval and generation are disabled; and the reviewer assesses quality rather than continuing the live chat. The reviewer may accept the conversation as is, accept it with edits, or reject it. Accept-with-edits permits refining responses, adjusting passage relevance, and modifying enrichments. Reviewers may not change the user questions or search for new passages, because doing so could alter the original conversational intent too much. Rejection is reserved for conversations that are too flawed to salvage without major changes, for example when questions are repetitive or unnecessary, conversation flow is poor, or available passages are insufficient. The interface also supports general comments and comments tied to highlighted spans (Fadnis et al., 22 Aug 2025).
Experiment mode serves as a lightweight real-time evaluation utility rather than a large-scale benchmark runner. Users upload conversation data, choose a conversation slice or task decomposition, select whether to evaluate generation alone or the full RAG pipeline, configure retriever and generator variants, and run built-in metrics. The reported supported metrics are response length, ROUGE, Recall, and LLM-as-a-Judge. The system can evaluate all turns, only the last turn, the beginning of a conversation, a random turn, or split at every turn so that each split becomes a task. Experiment outputs can be exported and loaded into InspectorRAGet for deeper metric and model analysis. The platform restricts experiment datasets to at most 100 tasks, which the authors interpret as a signal that this mode is intended for quick diagnostics and due diligence rather than large offline evaluation (Fadnis et al., 22 Aug 2025).
The absence of a formal scoring theory is explicit. The paper does not provide a benchmark loss, retrieval-scoring equation, or optimization objective; the evaluation framework is described as practical rather than mathematically formalized. The only mathematical notation reported is descriptive statistics from the user study, such as $\mu$ and $\sigma$ for means and standard deviations.
5. Data representation, scale of use, and empirical observations
Exported conversations are stored in structured JSON. According to the appendix, the JSON contains four broad sections: participant metadata such as author, editor, reviewer emails and timestamps; retriever and generator configuration details including connectivity and parameters; the actual message sequence with user turns, enrichments, assistant turns, and attached contexts or documents; and conversation status information such as revision history and reviewer comments (Fadnis et al., 22 Aug 2025).
The platform has been used at nontrivial scale. The abstract states that RAGAPHENE was successfully used by approximately 40 annotators to build thousands of real-world conversations. In the use-case section, the authors report using the creation and review workflows to produce 110 high-quality multi-turn RAG conversations released as the public MTRAG benchmark. They also report ongoing usage comprising over 5,000 conversations created and over 1,000 conversations reviewed by over 30 annotators (Fadnis et al., 22 Aug 2025).
The user study provides a more detailed empirical picture. The authors surveyed 31 professional annotators, of whom 21 were female and 10 male, and 13 had more than three years of annotation experience. The tool had been in production for more than nine months, and most annotators had created and reviewed over 75 conversations. Annotators reported that creating a single high-quality conversation usually took more than 30 minutes, indicating that realistic benchmark-grade multi-turn RAG data collection is labor-intensive (Fadnis et al., 22 Aug 2025).
Feature-ablation ratings in the study reveal which interface functions annotators considered most important. The highest-impact features, rated by how much their removal would decrease conversation quality on a 1–5 Likert scale, were editing the agent responses ($\mu = 4.26, \sigma = 0.82$), highlighting overlap between context and response ($\mu = 4.13, \sigma = 0.88$), regenerating the response ($\mu = 4.10, \sigma = 0.98$), and the requery tool ($\mu = 3.97, \sigma = 1.37$). The export checklist also scored fairly high ($\mu = 3.74$). Marking contexts relevant or irrelevant, hints, and enriching questions were rated lower. One annotator explicitly stated that marking contexts as relevant or irrelevant is useful to end-users of the data but not necessary for performing the creation task itself (Fadnis et al., 22 Aug 2025).
These findings support the platform’s central design choice: tools for repairing retriever and generator behavior were valued more highly than metadata-centric tasks. This suggests that, in practice, conversation quality depends most strongly on mechanisms that let annotators directly correct evidence selection and answer content.
6. Strengths, limitations, and prospective development
The paper presents several strengths. RAGAPHENE directly targets realistic multi-turn RAG benchmark construction by combining live retrieval-generation with human correction. It supports customizable retrievers, generators, and prompts, making it adaptable to enterprise and domain-specific settings. It addresses both retrieval-side and generation-side failures, includes a review pipeline for quality assurance, provides a lightweight experiment mode for rapid testing, and is described as modular, lightweight, and privacy-conscious through a stateless design. It has also already contributed to a released benchmark and been used at substantial annotation scale (Fadnis et al., 22 Aug 2025).
The limitations are also explicit. Experiment mode is capped at 100 tasks and is therefore not intended for large-scale evaluation. Conversation creation is time-consuming and cognitively demanding. Some annotation features appear less valued by users than the core correction tools, which suggests possible need for UI or workflow refinement. The authors also acknowledge accessibility limitations. In Review mode, the prohibition on changing user questions or searching for new passages preserves conversational intent, but it can also reduce the salvageability of partially flawed conversations. Finally, code release was still pending internal approval at the time of writing, so broader reproducibility and adoption depend on that release (Fadnis et al., 22 Aug 2025).
Future directions mentioned by the authors include releasing the code on GitHub under the Apache 2.0 license following internal approval, improving accessibility, and adding support for additional backends such as ChromaDB, Claude, and vLLM. More broadly, the work suggests continued development in scaling human-in-the-loop benchmark creation, improving annotation ergonomics, and tightening the connection between benchmark construction, model debugging, and downstream evaluation analytics (Fadnis et al., 22 Aug 2025).
In that sense, RAGAPHENE occupies a specific niche in the RAG tooling landscape: not a general-purpose chat interface, not only a benchmark runner, and not merely an annotation frontend, but a correction-oriented platform for constructing realistic, reviewable, experiment-ready multi-turn RAG benchmarks around live system behavior.