RAGTruth: Hallucination Benchmark for RAG Systems

• RAGTruth is a large-scale hallucination corpus designed to evaluate if generated content is strictly grounded in retrieved references.
• It features multi-task evaluations (QA, data-to-text, summarization) with detailed token- and span-level annotations.
• The benchmark drives advancements in detector models by exposing performance gaps across tasks and response lengths.

RAGTruth is a hallucination corpus and benchmark for retrieval-augmented generation (RAG) designed to measure whether generated content is supported by, or contradicts, the references supplied to the model. In its original presentation, it comprises nearly 18,000 naturally generated responses collected under standard RAG workflows and annotated at both the individual-case and word levels. Its central object of study is strict grounding: a span is treated as hallucinated when it is unsupported by, or conflicts with, the provided references, regardless of whether it might be true in the open world. This design has made RAGTruth a recurring benchmark for response-level, span-level, and token-level hallucination detection across question answering, data-to-text generation, and summarization (Niu et al., 2023).

1. Origin, motivation, and benchmark role

RAGTruth was introduced to address a specific failure mode of retrieval-augmented LLMs: even when an LLM is prompted with retrieved passages, structured records, or source documents, the generated answer may still contain unsupported additions or contradictions. The corpus was constructed to study these failures under realistic RAG prompting rather than under artificially synthesized factual conflicts. Its emphasis is therefore on naturally generated RAG outputs, fine-grained annotation, and task settings that reflect deployed retrieval-grounded systems (Niu et al., 2023).

Subsequent work has characterized RAGTruth as the first large-scale benchmark tailored to evaluating hallucinations in RAG settings and, to the best of some later authors’ knowledge, the only publicly available manually annotated dataset specifically designed for this purpose. Across that later literature, it functions as a backbone dataset for training detector models, comparing prompt-based judges with specialized classifiers, and probing whether detectors localize hallucinated spans or merely exploit dataset regularities (Kovács et al., 24 Feb 2025).

The benchmark’s importance also derives from its granularity. Later detector papers consistently rely on RAGTruth not only for answer-level labels but for span-level or token-level supervision, which enables models to mark unsupported fragments inside an otherwise partially faithful answer. This has made the dataset relevant to both diagnostic evaluation and deployment-oriented guardrails for RAG systems.

2. Corpus composition and task design

The original corpus contains 2,965 input instances, each paired with responses from six models, yielding nearly 18,000 responses in total. Later papers also describe the same resource as containing 17,790 model responses or “annotated examples at the span level,” reflecting response-level counting conventions over the same benchmark family (Niu et al., 2023).

Task	Source	Grounding setup
Question answering	MS MARCO	Daily-life question with three retrieved passages
Data-to-text writing	Yelp Open Dataset	Structured JSON fields plus up to three user reviews
News summarization	CNN/Daily Mail and recent news	Full article to be summarized under a word budget

In question answering, questions are randomly sampled from MS MARCO and matched with three retrieved passages. The answer is required to be based strictly on those passages, with a fallback instruction equivalent to “Unable to answer…” when the evidence is insufficient. In data-to-text generation, the inputs are Yelp businesses in restaurants and nightlife, represented through structured JSON fields such as BusinessParking, RestaurantsReservations, OutdoorSeating, WiFi, RestaurantsTakeOut, RestaurantsGoodForGroups, Music, and Ambience, together with up to three user reviews. In summarization, the sources are CNN/Daily Mail documents and recent news articles, with prompts imposing a word limit proportional to article length (Niu et al., 2023).

The six generating models listed in the original release are GPT-3.5-turbo-0613, GPT-4-0613, Mistral-7B-Instruct, Llama-2-7B-chat, Llama-2-13B-chat, and Llama-2-70B-chat. Later descriptions preserve the same multi-model structure and stress that each sample contains one response from each model, enabling model-level comparisons on identical prompts and evidence (Dubanowska et al., 19 Sep 2025).

Hallucination prevalence is highly task-dependent in the original report: 29.1% for QA, 68.6% for data-to-text, 30.9% for CNN/Daily Mail summarization, 27.6% for recent-news summarization, and 43.1% overall. The corpus also records strong length effects. As response length increases, hallucinations become more frequent across tasks; in QA and summarization they tend to occur toward the end of responses, whereas in data-to-text they are more frequent in the first half, especially around business attributes (Niu et al., 2023).

3. Annotation scheme, labels, and data structure

RAGTruth defines hallucination in strictly contextual terms: any generated span that is unsupported by, or contradicts, the provided references is labeled as hallucinated. The annotation taxonomy has four categories:

1. Evident Conflict
2. Subtle Conflict
3. Evident Introduction of Baseless Information
4. Subtle Introduction of Baseless Information

The evident/subtle distinction operationalizes intensity or severity, while conflict versus baseless information separates contradiction from unsupported addition. Later detector papers often collapse this four-way taxonomy into a binary label space—supported versus hallucinated—for supervised token classification, but the underlying span annotations retain the finer taxonomy (Niu et al., 2023).

The original annotation process used Label Studio. Each response was labeled independently by two annotators who were English-proficient and held bachelor’s degrees in relevant fields, with compensation reported as $25 per hour. Response-level agreement was 91.8%, span-level agreement was 78.8%, and a third review was used to resolve large discrepancies (Niu et al., 2023).

Two auxiliary span-level flags make the corpus adaptable to different evaluation philosophies. The implicit_true flag marks information not present in the references but plausibly true; in the benchmark’s strict RAG setting such spans are still hallucinations by default, but the flag enables alternative scoring conventions. The due_to_null flag is specific to data-to-text, where null JSON fields denote unknown rather than false; about 17.7% of hallucination spans in data-to-text are related to null values being misread as negative evidence (Niu et al., 2023).

At the record level, each example includes the RAG input, the model responses, and span annotations over the responses. For QA this means a question plus retrieved passages; for data-to-text, structured business data and reviews; for summarization, the source article. Later work describes the benchmark as monolingual in its original release and emphasizes that the reference materials—retrieved passages, source documents, or structured records—constitute the grounding target against which all response spans are verified (Taş et al., 22 Sep 2025).

4. Evaluation protocols and detector training

RAGTruth supports response-level and span-level evaluation. At the response level, the task is binary: determine whether a response contains any hallucination. At the span level, evaluation is based on character-level overlap between predicted and annotated hallucinated regions. The standard metrics reported in the original paper are precision, recall, and F1, with span-level precision and recall computed from the overlap of predicted versus gold hallucinated character sets (Niu et al., 2023).

Later work extends this evaluation regime to token-level classification. In that formulation, the model consumes a context–question–answer triple and predicts, for each answer token, whether it is supported or hallucinated. Context and question tokens are label-masked with label = -100 and excluded from the loss. The Turkish adaptation paper gives the binary cross-entropy objective explicitly as

$$L = -\sum_t \left[y_t \log p_t + (1-y_t)\log(1-p_t)\right],$$

where $y_t \in \{0,1\}$ is the gold token label and $p_t$ is the predicted probability for the hallucinated class; it also reports precision, recall, macro F1, and AUROC under this setup (Taş et al., 22 Sep 2025).

RAGTruth’s sequence lengths are a practical constraint for detector design. One later paper reports preprocessing statistics of mean length 801 tokens, median 741, minimum 194, and maximum 2,632, and cites this long-context characteristic as a direct motivation for ModernBERT-based encoders. In LettuceDetect, inputs are concatenated with special tokens, truncated to 4,096 tokens in the reported experiments, and used to train token classifiers whose predictions are aggregated into spans by thresholding answer-token hallucination probabilities above 0.5 (Kovács et al., 24 Feb 2025).

The dataset also serves as a benchmark for detector effectiveness. In the original paper, a fine-tuned Llama-2-13B detector trained on RAGTruth achieved overall response-level precision 76.9, recall 80.7, and F1 78.7, compared with GPT-4-turbo prompt-judging at response-level F1 63.4. At span level, the same fine-tuned detector reached precision 55.6, recall 50.2, and F1 52.7, compared with GPT-4-turbo at F1 28.3. These results were used to argue that high-quality RAG-specific supervision can outperform large prompt-based judges for hallucination detection (Niu et al., 2023).

5. Re-annotation, confounds, and benchmark critique

Although RAGTruth became a standard benchmark, later work has argued that its original labels understate hallucination prevalence. RT4CHART introduces two re-annotated resources derived from the evaluation split: RAGTruth++, a 408-example QA/summarization subset with independent dual-annotator review, and RAGTruth-Enhance, a 2,675-example re-annotation of the full evaluation split. On the 408-example subset, the number of annotated hallucination spans increases from 86 to 865. Across the full evaluation split, the re-annotation uncovers 1.68 times more hallucination cases and 3.1 times more hallucination spans than the original labels (Yu et al., 29 Mar 2026).

The same work categorizes 3,540 newly added hallucination spans in changed cases into recurring error patterns: Unsupported Generalization (38.62%), Numeric/Logic Inconsistency (25.42%), Inference Stated as Fact (17.54%), Prior Knowledge Interference (8.08%), and Others (10.34%). These categories suggest that subtle micro-hallucinations were frequently left unlabeled in the original release, especially when the generated text remained plausible on its surface (Yu et al., 29 Mar 2026).

A distinct critique targets benchmarking methodology rather than annotation completeness. The paper on representation-based detectors argues that aggregated performance on RAGTruth can be driven by spurious correlation with task type, especially the data-to-text subset with JSON-formatted prompts and markedly higher hallucination rates. For LLaMA-2-7B-Chat, a naïve classifier that predicts hallucination for all data-to-text instances and non-hallucination otherwise achieves AUC 0.7119 and PCC 0.4494, rivaling or exceeding several broad detectors. The same study reports that cross-task and cross-dataset generalization often collapses toward random performance, indicating reliance on task- or format-specific artifacts rather than robust hallucination signals (Dubanowska et al., 19 Sep 2025).

These critiques do not negate RAGTruth’s utility, but they materially affect how it is interpreted. One implication drawn in the later literature is that benchmark reporting should separate QA, data-to-text, and summarization rather than relying only on aggregate scores. Another is that re-annotation, out-of-distribution evaluation, and span-level localization are necessary if detector performance is to be interpreted as evidence of genuine grounding awareness rather than exploitation of label imbalance or prompt structure.

6. Extensions, multilingual adaptations, and continuing use

RAGTruth has been extended beyond its original English release through structurally faithful translation pipelines. The Turkish work "Turk-LettuceDetect" describes a machine-translated Turkish version of RAGTruth in which hallucination spans are preserved via explicit <HAL>...</HAL> tagging during translation. The translated files mirror the English schema, retain split and type fields, add updated character-level indices (start, end, label), and include a language: tr field. That paper reports a training split of 17,790 instances and a test split of 2,700 instances for detector fine-tuning and evaluation, with no development split reported (Taş et al., 22 Sep 2025).

The benchmark also remains a primary in-distribution testbed for newer detector families. A proxy-analyzer study uses RAGTruth as its primary benchmark for token-level hallucination detection and reports that RAGTruth contributes 15,090 samples within a broader 72,135-sample multi-dataset training pool, using a stratified 70/15/15 split and a specialist classifier trained on 10,563 RAGTruth training rows. Across seven analyzer architectures, that work reports token-level AUCs that consistently exceed ReDeEP’s token-level AUC of 0.733 on RAGTruth (Singh et al., 8 May 2026).

More recent perturbation-based work narrows the benchmark to settings with sufficiently long answers for stable span estimation. GASP evaluates on a class-balanced 400-response subset drawn from RAGTruth summarization and data-to-text, under a leakage-clean protocol that groups all spans from the same response into the same fold. On that subset, it reports response-level AUC values around 0.73 and span-level AUC values around 0.67, outperforming perplexity, length, whole-context NLI, and self-consistency baselines, while remaining competitive with a chunk-level entailment verifier at span level (Bouke, 5 Jul 2026).

Across these extensions, RAGTruth has evolved from a single corpus into a benchmark family with translated variants, re-annotated derivatives, and specialized evaluation subsets. Its enduring contribution is the combination of multi-task RAG inputs, multi-model outputs, and fine-grained human annotations. Its principal limitations are also now well documented: strict context-only labeling can mark true but unsupported claims as hallucinations, annotation completeness can matter as much as detector design, and aggregate scores can be misleading when task-type confounds are not controlled.