Faithfulness-QA: Context-Faithful Retrieval Dataset

• Faithfulness-QA is a large-scale dataset that benchmarks context-faithfulness by creating explicit knowledge conflicts through counterfactual entity substitution.
• It employs an automated counterfactual entity substitution pipeline that replaces named entities while preserving most of the original context to induce controlled deviations.
• The dataset serves as both a training resource for attention-based faithfulness objectives and an evaluation benchmark to differentiate context-grounded from memory-based responses.

Faithfulness-QA is a large-scale dataset for context-faithful retrieval-augmented generation (RAG) that makes knowledge conflict explicit by modifying extractive QA examples so that the answer supported by the provided context differs from the answer a model is likely to retrieve from parametric memory. It contains 99,094 samples constructed through counterfactual entity substitution, starting from SQuAD and TriviaQA, automatically identifying answer-bearing named entities and replacing them with type-consistent alternatives drawn from a curated bank of 76,953 entities (Ju et al., 28 Apr 2026). Its purpose is twofold: it is designed as a training resource for attention-based faithfulness objectives and as an evaluation benchmark for measuring context-grounding behavior in RAG systems.

1. Conceptual role in Faithfulness-QA research

Faithfulness-QA is motivated by a specific failure mode of retrieval-augmented generation: even when a relevant context passage is retrieved and prepended to the model input, the model may still answer from pretrained parametric memory rather than from the retrieved evidence. The paper frames this as a structural issue in standard language-model training, where ordinary next-token prediction does not explicitly teach the model that, when retrieved passages conflict with memorized knowledge, the passage should be preferred (Ju et al., 28 Apr 2026).

The dataset makes this failure observable by introducing knowledge conflict. In ordinary QA examples, context and parametric memory usually agree, so a correct answer does not reveal whether the model actually used the context. Faithfulness-QA changes that condition. After substitution, the modified context supports a new answer, while the original real-world answer remains the likely memory-based response. Under this setup, a model is faithful if it outputs the answer implied by the modified context, and unfaithful if it outputs the original answer despite contradictory contextual evidence (Ju et al., 28 Apr 2026).

This suggests a narrow but important operationalization of faithfulness: not generic factuality, but context-grounding under conflict. That makes the dataset particularly suited to RAG systems whose central promise is not merely accuracy, but adherence to retrieved evidence.

2. Counterfactual entity substitution pipeline

Faithfulness-QA starts from a standard extractive QA triple,

$(q, c, a),$

where $q$ is the question, $c$ is the context, and $a$ is the answer. The pipeline then identifies an answer-bearing named entity in the context and replaces it with another entity of the same type. The question remains unchanged. The modified context therefore induces a new faithful answer while preserving most of the passage structure (Ju et al., 28 Apr 2026).

Answer-entity matching is performed with a three-stage cascade: exact match, substring match, and positional overlap. If no entity can be matched, the sample is discarded. Once an original answer-bearing entity $(e_{\text{orig}}, t)$ is identified, the system samples a replacement entity $e_{\text{new}}$ from the entity bank subject to three constraints: the replacement must have the same type,

$\text{type}(e_{\text{new}})=t,$

it must differ from the original entity, and it must satisfy the approximate length compatibility condition

$$0.3 \leq |e_{\text{new}}|/|e_{\text{orig}}| \leq 3.0.$$

The modified context is then created by global replacement,

$$c' \leftarrow ReplaceAll(c, e_{\text{orig}}, e_{\text{new}}),$$

and the faithful answer is set to

$a' = e_{\text{new}}.$

These steps are all described as fully automated and deterministic with random seed 42 (Ju et al., 28 Apr 2026).

The intervention targets named entities because they are easy to identify automatically with NER, frequently serve as answer-bearing spans in factoid QA, and permit type-consistent substitution. The paper emphasizes that this yields a controlled modification: the context changes minimally but decisively, creating a conflict between retrieved evidence and likely pretrained knowledge without rewriting the entire example (Ju et al., 28 Apr 2026).

3. Entity bank, typing system, and quality controls

A central artifact of Faithfulness-QA is its typed entity bank of 76,953 unique entities, extracted from all 19,035 unique SQuAD contexts using spaCy’s en_core_web_lg NER model. The bank covers eight named-entity categories: ORG, PERSON, DATE, GPE, CARDINAL, LOC, NORP, and EVENT. The reported composition is: ORG 25,378 (33.0%), PERSON 20,292 (26.4%), DATE 10,613 (13.8%), GPE 6,769 (8.8%), CARDINAL 6,636 (8.6%), LOC 2,977 (3.9%), NORP 2,849 (3.7%), and EVENT 1,439 (1.9%) (Ju et al., 28 Apr 2026).

Quality control is implemented at two levels. During construction, each candidate must pass six quality checks: the replacement entity must appear in the modified context; the modified context must differ from the original; the modified context must be at least 50 characters long; the replacement entity must not already appear in the original context; the original entity must occur at most 10 times in the original context; and the replacement entity must be at least 2 characters long (Ju et al., 28 Apr 2026).

The paper also reports a post hoc structural audit using a 200-sample random audit from the SQuAD subset. This audit verifies four automated properties: the replacement entity appears in $q$0; the original entity does not appear in $q$1; the context changed; and the length ratio remains within $q$2. All four checks achieved 200/200 (100%), and the aggregate “All checks” score is 100% (Ju et al., 28 Apr 2026).

These checks establish strong structural reliability, but the paper is explicit that they do not guarantee full semantic naturalness. It acknowledges several residual noise sources: no coreference resolution, no NLI-based semantic consistency filtering, syntactically valid but semantically implausible substitutions, and NER-matching limitations for descriptive or non-entity answers (Ju et al., 28 Apr 2026).

4. Scale, schema, and benchmark structure

The final dataset contains 99,094 samples produced from 174,640 source QA pairs, for an overall success rate of 56.7%. The source-specific breakdown is 49,094 output samples from 87,599 SQuAD inputs with success rate 56.0%, and 50,000 output samples from 87,041 TriviaQA inputs with success rate 57.4% (Ju et al., 28 Apr 2026).

Source	Output samples	Success rate
SQuAD	49,094	56.0%
TriviaQA	50,000	57.4%
Total	99,094	56.7%

The dataset is split 80/10/10 into 79,275 train, 9,909 dev, and 9,910 test examples. Stored in JSONL format at approximately 367 MB, each instance contains ten fields: id, question, original_context, modified_context, original_answer, faithful_answer, original_entity, replacement_entity, entity_type, and source (Ju et al., 28 Apr 2026).

The final entity-type distributions differ by source. SQuAD-derived counterfactuals are relatively balanced, while TriviaQA-derived counterfactuals are heavily PERSON-dominant. Context lengths are reported in characters: for SQuAD-derived examples, mean 763, median 701, min 151, max 3,706; for TriviaQA-derived examples, mean 1,313, median 1,581, min 108, max 2,122. The filtering analysis attributes most discarded cases to No NER entity match: 48,412 (64.0% of filtered cases), followed by No context available: 13,272 (17.6%) and No replacement in bank: 12,064 (16.0%) (Ju et al., 28 Apr 2026).

The paper does not report downstream model baselines or benchmark scores. It explicitly states that there is no downstream evaluation, and positions the work as a dataset construction and analysis paper rather than a benchmark-results paper (Ju et al., 28 Apr 2026).

5. Intended training and evaluation uses

Faithfulness-QA is intended for faithfulness-aware fine-tuning, attention-based faithfulness loss, knowledge conflict research, and evaluation. The core supervised use is straightforward: treat modified_context as the retrieved passage, keep the original question, and train the model to output faithful_answer (Ju et al., 28 Apr 2026).

As an evaluation benchmark, the dataset provides both original_answer and faithful_answer, making it possible to distinguish two behaviors directly. A model that outputs the replacement entity is context-grounded; a model that outputs the original answer reveals reliance on parametric memory rather than the modified context. This direct separation is the benchmark’s main methodological advantage (Ju et al., 28 Apr 2026).

The paper also proposes a more specialized use in attention-based faithfulness objectives. It suggests that the explicit modified region could supervise cross-attention weights so that models attend to the answer-bearing context rather than defaulting to memory. However, it does not provide an explicit loss equation or an experimental implementation for such an objective (Ju et al., 28 Apr 2026).

This design aligns naturally with broader Faithfulness-QA evaluation principles in retrieval-augmented systems. Work on instruction-following QA distinguishes correctness from faithfulness w.r.t. provided knowledge, and proposes evidence-conditioned metrics such as K-Precision for whether model responses stay grounded in passages (Adlakha et al., 2023). Work on RAG fact-checking likewise argues that faithfulness and factuality should be separated, because unsupported statements can still be true in the world (Fadeeva et al., 27 May 2025). Faithfulness-QA is narrower: it isolates the case where the supplied context is the authority, and tests whether the model follows it under conflict (Ju et al., 28 Apr 2026).

6. Position within the broader faithfulness literature

Faithfulness-QA occupies a specific place in the rapidly growing literature on contextual faithfulness. It is not itself a decoding method, a reward model, or an activation-steering technique. Instead, it supplies the controlled conflict data that such methods often require but usually lack (Ju et al., 28 Apr 2026).

This makes it directly relevant to work on situated faithfulness under incorrect or conflicting contexts, where models must arbitrate between internal knowledge and external evidence (Huang et al., 2024). It is also closely aligned with research on contextual faithfulness under knowledge conflict, including activation steering for following counterfactual or updated context rather than memorized facts (Anand et al., 7 Jan 2026). A plausible implication is that Faithfulness-QA can serve as a standardized training or stress-testing resource for such conflict-sensitive methods, because it turns hidden reliance on memory into an explicit, measurable error condition.

The dataset also complements approaches that improve contextual faithfulness through self-evolution or reinforcement learning. Sentence-level self-evolution for long-form QA seeks to reduce unsupported details at the sentence level (Li et al., 3 Mar 2025), while synthetic-task and reinforcement-learning pipelines train models to remain faithful to supplied context across short-form and long-form generation without human annotations (Si et al., 22 May 2025). Faithfulness-QA differs in function: it does not itself optimize models, but it provides a large-scale supervision source where the desired behavior—prefer context over memory—is unambiguous (Ju et al., 28 Apr 2026).

Finally, the paper contrasts Faithfulness-QA with earlier resources. Standard extractive QA datasets such as SQuAD and TriviaQA do not systematically create context-memory conflict. FaithEval is described as an evaluation dataset with only 4,900 samples, not a large-scale training resource. CounterFact and MQuAKE target knowledge editing rather than context-faithful retrieval use. KAFT includes counterfactual and irrelevant contexts for controllability and robustness, but Faithfulness-QA’s novelty lies in a dedicated dataset centered on context-vs-memory conflicts in QA examples (Ju et al., 28 Apr 2026).

7. Limitations, caveats, and research significance

The paper is explicit about several limitations. First, the approach is restricted to named entities, which excludes many QA examples with descriptive, abstract, common-noun, or boolean answers. This is visible in the dominant skip reason, No NER entity match, which accounts for 64.0% of filtered cases (Ju et al., 28 Apr 2026). Second, the modifications are synthetic, and the paper acknowledges that some substitutions preserve syntax while breaking world plausibility. Third, the dataset does not model all real RAG failure modes, such as retrieval ranking errors, evidence fragmentation across passages, or conflicting information across multiple retrieved sources (Ju et al., 28 Apr 2026).

The paper also notes that Faithfulness-QA is not a full-stack RAG benchmark. It gives the model a directly modified context, which is valuable for measuring context obedience, but does not simulate the entire retrieval pipeline. That makes the dataset especially suitable for generator-side faithfulness analysis and training, but less informative about retrieval quality itself (Ju et al., 28 Apr 2026).

Even with these caveats, the dataset’s significance is clear. It provides foundational infrastructure for a central open problem in retrieval-augmented generation: how to distinguish models that merely answer correctly from models that answer because they followed the provided evidence. By manufacturing controlled knowledge conflicts at scale, Faithfulness-QA makes that distinction observable, trainable, and benchmarkable (Ju et al., 28 Apr 2026).