RAGTruth Corpus: Hallucination Benchmark

• RAGTruth Corpus is a manually annotated dataset focusing on unsupported and contradictory content in retrieval-augmented generation using strict evidential criteria.
• It covers key tasks including question answering, data-to-text writing, and news summarization with 17,790 responses across 2,965 instances.
• The corpus enables detailed error detection at both response and span levels, supporting improved evaluation and suppression of hallucinations.

RAGTruth is a manually annotated hallucination corpus for retrieval-augmented generation (RAG) that was designed to measure unsupported and contradictory content relative to the retrieved or otherwise provided reference context, rather than relative to unrestricted world knowledge. It targets word-level hallucination analysis in standard RAG workflows and spans question answering, data-to-text writing, and news summarization. The corpus contains 2,965 instances and 17,790 naturally generated responses from six instruction-following LLMs, with response-level and span-level annotations that distinguish multiple forms of conflict and unsupported addition (Niu et al., 2023).

1. Scope and evidential definition

RAGTruth is defined around a strict evidential notion of truthfulness. In this setting, a response is judged against the passages, articles, or structured records supplied to the model, not against broad external knowledge. The corpus therefore treats two broad phenomena as hallucination: unsupported claims and contradictory claims. Unsupported claims are details not substantiated by the provided reference content; contradictory claims directly conflict with that content (Niu et al., 2023).

This definition is deliberately stricter than generic factuality assessment. The corpus explicitly counts information “that is not mentioned in the reference but may be truthful” as hallucination, because the intended RAG use case requires models to answer only from supplied evidence. A common misconception is therefore that a statement must be globally false to count as hallucination. In RAGTruth, that is not the criterion. A claim may be true in the world and still be labeled hallucinatory if it is unsupported by the retrieved context (Niu et al., 2023).

The paper further refines hallucinations into four categories. Evident Conflict denotes direct contradiction or opposition to provided information. Subtle Conflict denotes a more nuanced divergence that changes contextual meaning. Evident Introduction of Baseless Information denotes fabricated or hypothetical details with no support in the reference. Subtle Introduction of Baseless Information denotes inferential, subjective, or norm-based additions not explicitly verifiable from the source. This taxonomy operationalizes both error type and intensity, with “evident” versus “subtle” functioning as the severity distinction (Niu et al., 2023).

2. Corpus construction and task coverage

The corpus was built from a generation-and-annotation pipeline covering three tasks: question answering, data-to-text writing, and news summarization. For question answering, the authors randomly sampled from the training set of MS MARCO, retained daily-life questions, and preserved only three retrieved passages per question. For data-to-text writing, they used the Yelp Open Dataset, restricted businesses to the restaurant and nightlife categories, represented core business fields in JSON, and added up to three user reviews. For summarization, they sampled from the CNN/Daily Mail training set and from recent news articles from “a prestigious news platform” (Niu et al., 2023).

The generation prompts were task-specific and evidence-constrained. Question answering instructed the model to answer strictly from three passages and to reply with “Unable to answer based on given passages.” if necessary. Data-to-text writing instructed the model to write an objective 100–200 word overview based only on the provided structured data and reviews, with “Don’t make up information.” Summarization used a target length of $\min(200, \text{word\_num\_of\_news} // 4)$; the paper states that this length control was not used as a hallucination criterion (Niu et al., 2023).

Each input instance received one response from each of six instruction-following LLMs: GPT-3.5-turbo-0613, GPT-4-0613, Mistral-7B-Instruct, Llama-2-7B-chat, Llama-2-13B-chat, and a 4-bit quantized Llama-2-70B-chat. This yields six responses per instance across all tasks (Niu et al., 2023).

Task	Instances	Context form
Question answering	989	Question + three retrieved passages
Data-to-text writing	1,033	Business JSON + up to three reviews
News summarization	943	Source news article

These 2,965 instances produce 17,790 responses overall. The abstract describes the resource as “nearly 18,000 naturally generated responses,” while the paper gives the exact count implied by the instance-by-model construction (Niu et al., 2023).

3. Annotation protocol and label structure

RAGTruth’s defining feature is meticulous manual annotation at both the individual-case level and the word or span level. Annotators marked specific spans of generated text containing hallucinated information and assigned each such span to one of the four hallucination categories. Each response also received a response-level judgment indicating whether it contained hallucination (Niu et al., 2023).

The annotation scheme includes two auxiliary span labels that materially affect downstream use. implicit_true marks spans that are not in the evidence but may nevertheless be true. The authors still count these spans as hallucinations under the strict RAG definition, but expose the flag so that users can adopt a looser evaluation regime if desired. due_to_null marks data-to-text spans where a model interpreted a JSON null value as negation; the paper treats this as hallucination because null denotes unknown rather than false (Niu et al., 2023).

The annotation process used Label Studio. Each annotation page presented the context and six model responses. Every response was independently labeled by two annotators, and a third review was performed when there was a “considerable difference” between the two annotations. Annotators were recruited through a professional vendor, were proficient in English, held bachelor’s degrees in English, Communications, or related fields, and were paid \$25 per hour per person. Reported agreement was 91.8% at the response level and 78.8% at the span level (Niu et al., 2023).

The corpus supports both strict and adaptive evaluation. Because the labels distinguish unsupported-but-possibly-true content and null-handling errors, the same underlying data can serve applications that require rigid evidence faithfulness and applications that prefer a looser notion of factual acceptability. The paper does not introduce a numeric hallucination intensity score; instead, intensity is encoded categorically through the evident/subtle distinction (Niu et al., 2023).

4. Empirical characteristics of hallucination in RAGTruth

RAGTruth reports that 43.1% of responses contain hallucinations overall. The rate varies substantially by task. Data-to-text writing is the most hallucination-prone setting by a wide margin, whereas question answering and summarization are markedly lower. Within summarization, the paper distinguishes CNN/Daily Mail and recent-news subsets analytically (Niu et al., 2023).

Setting	Hallucination rate
Question Answering	29.1%
Data-to-text Writing	68.6%
Summarization (CNN/DM)	30.9%
Summarization (Recent News)	27.6%
Overall	43.1%

The paper attributes the especially high data-to-text rate partly to inconsistent handling of JSON-format data, especially time fields and attributes. It also reports that recent-news summarization did not produce more hallucinations than older/newswire summarization, contrary to an expectation that parametric knowledge staleness would dominate. The authors suggest that the recent-news contexts may have been shorter than CNN/Daily Mail contexts (Niu et al., 2023).

Hallucination type distributions indicate that baseless information is much more common than conflict, especially in question answering. The paper also states that the more severe categories—Evident baseless info and Evident conflict—account for a significant portion of errors. This suggests that many failures are not merely borderline inferences but clear unsupported additions or clear contradictions (Niu et al., 2023).

Model-wise, GPT-4-0613 has the lowest hallucination frequency and density, and GPT-3.5-turbo-0613 is next best among the listed models. Among the Llama-2 variants, larger scale generally correlates with lower hallucination density, though not consistently in data-to-text writing. Mistral-7B-Instruct has the highest number of hallucinating responses in the benchmark (Niu et al., 2023).

The corpus analysis also examines length and position effects. Longer responses contain more hallucinated spans across all three tasks. Context length matters strongly only for summarization, where longer source articles lead to more hallucinations. In question answering and summarization, hallucinations are more likely near the end of the response; in data-to-text writing, they occur relatively more often in the first half, especially around business attributes. The paper further reports that about 17.7% of hallucination spans in data-to-text writing are related to null values in the JSON data (Niu et al., 2023).

5. Benchmark use for detection, localization, and suppression

RAGTruth was used to benchmark both response-level hallucination detection and span-level localization. The evaluated approaches were prompt-based detection with GPT-3.5-turbo and GPT-4-turbo, SelfCheckGPT, LMvLM, and a fine-tuned Llama-2-13B trained on the RAGTruth training split. For fine-tuning, the model used context-response pairs plus instructions as input and hallucinated spans as targets, with full fine-tuning, initial learning rate $2e^{-5}$, 1 epoch, and 4 A100 GPUs (Niu et al., 2023).

At the response level, the fine-tuned Llama-2-13B substantially outperformed the prompt-based methods. Overall F1 scores were 52.9 for prompt GPT-3.5-turbo, 63.4 for prompt GPT-4-turbo, 58.8 for SelfCheckGPT with GPT-3.5-turbo, 49.4 for LMvLM with GPT-4-turbo, and 78.7 for fine-tuned Llama-2-13B. By task, the fine-tuned model achieved F1 scores of 68.2 on question answering, 88.1 on data-to-text writing, and 59.1 on summarization (Niu et al., 2023).

Span-level localization was substantially harder. Overall span-level F1 was 12.8 for the GPT-3.5 prompt baseline, 28.3 for the GPT-4 prompt baseline, and 52.7 for fine-tuned Llama-2-13B. The paper emphasizes that even GPT-4-turbo had only 18.4 precision overall at span level, whereas the fine-tuned model produced much stronger localization performance. It also reports that evident hallucinations were easier to detect than subtle ones, and that subtle conflict was the hardest category (Niu et al., 2023).

The paper then uses the detector operationally for hallucination suppression. On 450 test instances, it compared random selection of candidate outputs with detector-guided selection. For the more hallucination-prone pair Llama-2-7B-chat and Mistral-7B-Instruct, the hallucination rate was 52.4 under random selection, 41.1 when selecting the response with fewer detected hallucination spans, and 19.3 when selecting a response with no detected hallucination spans if available. For GPT-3.5-Turbo-0613 and GPT-4-0613, the corresponding rates were 9.8, 5.6, and 4.8. This shows that RAGTruth can support not only evaluation, but also reranking and filtering pipelines (Niu et al., 2023).

In practical terms, the corpus supports binary response-level detection, span extraction, hallucination type classification, severity-aware evaluation, and strict-versus-loose evaluation regimes based on the auxiliary labels. These use cases follow directly from the corpus structure: task-specific context, generated answer, response-level hallucination status, hallucinated spans, four-way type labels, and auxiliary flags (Niu et al., 2023).

6. Position within the RAG evaluation ecosystem

RAGTruth occupies a specific niche within RAG evaluation: fine-grained, human-annotated hallucination analysis. A later benchmark paper characterizes it succinctly as follows: “RAGTruth combines QA, Data-toText, and Summarization RAG data with human annotated hallucinated spans in the response. While it is an excellent benchmark for hallucination detection, it does not offer the level of granularity we present with RAGBench that is necessary to understand the RAG system as a whole.” In that comparison, RAGBench extends evaluation to retrieval-side and coverage-side signals through TRACe—Utilization, Relevance, Adherence, and Completeness—whereas RAGTruth remains centered on hallucination spans in generated responses (Friel et al., 2024).

This distinction is methodologically important. RAGTruth is strongest where the objective is to detect and localize unsupported or contradictory content in outputs under a strict evidence-grounded definition. It is less explicitly designed to measure retrieval adequacy, context utilization, or completeness as separate axes. A plausible implication is that RAGTruth and RAGBench are complementary rather than substitutable: one provides human gold hallucination supervision, while the other provides broader explainable system-evaluation signals (Friel et al., 2024).

Subsequent infrastructure work further clarifies the surrounding ecosystem. “RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits” does not explicitly mention the RAGTruth Corpus, and it is not a corpus release paper. It presents a chat-based production environment for creating and reviewing multi-turn RAG conversations with human correction of retrieved passages and generated answers. A plausible implication is that such tooling documents the kind of annotation infrastructure that could support closely related human-created RAG benchmarks, but it does not establish the origin of RAGTruth by name (Fadnis et al., 22 Aug 2025).

RAGTruth’s main limitations are also part of its historical significance. The paper presents it as a first benchmark attempt specifically for RAG hallucination and notes that rapidly evolving LLM applications may expose situations not covered by the current design. The described scope boundaries include three task types, six generation models, one specific retrieval setup for question answering, a restricted set of business attributes for data-to-text writing, and a strict evidential criterion that some applications may regard as overly harsh. Those constraints do not diminish its role as a core benchmark; rather, they define the regime in which its annotations are most interpretable and most valuable (Niu et al., 2023).