ContraTales Benchmark

• ContraTales is a synthetic benchmark comprising 2,000 narrative examples with clear constraints to test contextual logical consistency.
• The dataset pairs a story prefix with one correct ending and one contradictory ending, forcing models to evaluate internal narrative coherence.
• Empirical findings show that a single linear residual probe effectively distinguishes faithful from hallucinated endings, outperforming surface heuristics.

ContraTales is a 2,000-example benchmark introduced to evaluate contextual hallucination detection under especially hard, purely logical contradictions. Each example is a short story in which the prefix establishes a clear constraint in its first sentence, and two alternative concluding sentences are provided: one correct and one contradictory. The benchmark targets intrinsic contextual hallucinations, defined here as statements that contradict or are unsupported by the provided source context rather than by external world knowledge. In the associated study, ContraTales functions both as an evaluation resource for post-hoc hallucination detection and as a mechanistic interpretability testbed for analyzing whether a single linear direction in an observer model’s residual stream can separate faithful from hallucinated continuations (O'Neill et al., 31 Jul 2025).

1. Definition and task formulation

ContraTales is designed so that the context itself fully determines whether the ending is valid. The beginning of each tale sets up a simple, unambiguous constraint, such as “Jack had been bald for 10 years,” “Sarah was allergic to peanuts,” “Daniel can’t swim,” or “Laura didn’t own a smartphone.” A correct ending respects that constraint, while a contradictory ending logically violates it. The benchmark therefore operationalizes contextual hallucination as logical inconsistency between the story prefix and the final sentence, with no need for external fact checking or subjective interpretation (O'Neill et al., 31 Jul 2025).

This formulation distinguishes ContraTales from many hallucination benchmarks, especially in summarization, that emphasize factual or reference-based discrepancies and may be vulnerable to simple heuristics such as n-gram overlap. ContraTales is synthetic, but its contradictions are intentionally constructed to reuse entities and vocabulary rather than relying on lexical novelty. Lexical novelty, entity novelty, and even basic semantic similarity are therefore weak signals. This makes the benchmark a stress test for detectors that must reason over the internal logic of a document rather than over surface artifacts.

The benchmark is also explicitly restricted to intrinsic hallucinations. In this setting, the task is not to determine whether a sentence is true in the world, but whether it is consistent with the story that precedes it. A plausible implication is that ContraTales isolates a narrower but cleaner failure mode than broader hallucination datasets: logical contradiction inside a bounded textual context.

2. Dataset construction and annotation structure

ContraTales contains 2,000 examples. Each example consists of three textual components: a story prefix, a correct concluding sentence, and a contradictory concluding sentence. The prefix is a generated narrative of typically 7–10 sentences. Its first sentence states the governing constraint, and the following sentences develop an everyday scene while crucially avoiding repetition or negation of that initial constraint. The two alternative endings are each a single sentence, matched in tone and style, with one coherently continuing the story and the other introducing a detail that logically contradicts the initial condition (O'Neill et al., 31 Jul 2025).

The labels are deliberately unambiguous. A concluding sentence is faithful if it is a logical and coherent continuation that does not violate the first-sentence constraint. It is hallucinated if it introduces a detail that logically contradicts that constraint. Because the constraints are intentionally simple and binary, the benchmark avoids dependence on external knowledge and minimizes interpretive ambiguity. For each prefix, the dataset therefore yields one positive example and one negative example.

The data are wholly synthetic. Base story prefixes were initially generated by Claude Opus, while correct and contradictory endings were generated by o4-mini. Generation was controlled through a detailed prompt specifying the desired structure, the need for subtle logical contradictions, and the avoidance of lexically obvious cues. The prompt included few-shot examples, specifically 5 out of a seed set of 24, to steer the models toward the intended pattern. The domain is narrative rather than news or question answering, which matters because the benchmark’s central mechanism is the maintenance and violation of local story constraints rather than reference to external documents or world facts.

The use of short tales is methodologically important. Narrative structure makes it natural to establish a stable premise early and then produce an ending that either respects or violates it. This suggests that the benchmark is engineered to test whether a model can preserve latent contextual constraints over several sentences and then evaluate a candidate continuation against them.

3. Evaluation protocol and probe mechanics

ContraTales is used primarily for sequence-level binary classification. For each example and each ending, the evaluation constructs a source context $X$ from the story prefix and a continuation $Y$ from one of the two candidate concluding sentences. These are concatenated and fed to the observer model as a single sequence $(x_0,\dots,x_{T-1})$. The label is $y=1$ for a hallucinated ending and $y=0$ for a faithful ending. There is no token-level labeling in the benchmark (O'Neill et al., 31 Jul 2025).

All detectors are evaluated with logistic regression classifiers under 5-fold cross-validation. Training folds are used for fitting the classifier and, for probes, selecting the optimal layer; the held-out fold is used for evaluation. The main metric is F1 score on the binary classification task. The paper reports mean F1 with variability across folds, or bootstrap confidence intervals on plots. Baselines such as lexical overlap, entity verification, semantic similarity, and Lookback Lens are placed under the same cross-validation regime.

The residual-stream detector uses a frozen observer transformer $F$ with layers $l=0,\dots,L-1$. Let $t^*=T-1$ denote the final token of the continuation’s final sentence, and let $r^{(l)}_t\in\mathbb{R}^d$ be the post-layer-norm residual stream. For a chosen layer $l^*$, the probe input is

$Y$0

A logistic probe with parameters $Y$1 and $Y$2 computes

$Y$3

The default decision rule predicts hallucination if $Y$4, equivalently if $Y$5. The scalar logit $Y$6 is also used as a continuous hallucination score for analysis and attribution.

Training uses binary cross-entropy with L2 regularization:

$Y$7

Within ContraTales, each $Y$8 comes from either a correct or contradictory ending attached to a given prefix. The key methodological claim is that a single linear direction, represented by $Y$9, can separate faithful and hallucinated endings in the residual stream.

4. Role in the paper’s experimental program

ContraTales occupies three distinct roles in the broader study. First, it is a hard logical-contradiction benchmark for evaluating whether a linear residual probe can detect contextual inconsistencies that are not reducible to surface factual discrepancies. Second, it functions as an out-of-domain test relative to CNN/DailyMail and XSum, since news summarization and short-story narratives differ substantially in style and error structure. Third, it is used for domain adaptation experiments testing whether unsupervised finetuning of the observer on correct ContraTales stories sharpens the internal hallucination signal (O'Neill et al., 31 Jul 2025).

The observer models include GPT-2-small and Gemma-2 variants at 2B, 9B, and 27B, with the 27B model simulated in some plots. In the primary detection setting, these observer models are frozen. The procedure is to concatenate prefix and ending, perform a single forward pass, extract residual-stream activations at every layer for the final token of the last sentence, train a logistic regression probe at each layer, and compare layerwise performance against baselines and attribution analyses. In the adaptation setting, the observer is finetuned on 1,000 correct-only ContraTales completions and the probe is retrained on labeled data.

This integration is conceptually important because ContraTales is not used to measure raw generation hallucination rates. The endings are pre-generated. Instead, the benchmark evaluates whether contextual contradiction is linearly represented inside model activations, whether that representation transfers across domains and model sizes, and whether it can be improved by in-domain language modeling without contradiction labels.

5. Empirical findings on ContraTales

The reported results position ContraTales as a difficult benchmark for surface-based heuristics and a favorable setting for residual-stream probing. On ContraTales, Gemma-2-9B probes reach approximately 0.70 F1 at their best layers, while Gemma-2-27B, simulated in some plots, reaches F1 = 0.84. Layerwise performance rises in early layers and plateaus in mid-to-late layers, mirroring the structure observed on news datasets. The paper interprets this as evidence that a single linear direction emerges in the residual stream and carries a hallucination-versus-faithful signal (O'Neill et al., 31 Jul 2025).

Relative to baselines, the linear residual-stream probe achieves approximately $(x_0,\dots,x_{T-1})$0 F1 on ContraTales. The reported baseline values are lexical overlap at approximately 0.66 F1, entity verification at approximately 0.55 F1, and Lookback Lens at approximately $(x_0,\dots,x_{T-1})$1 F1. The probe therefore outperforms baselines by 9–27 points on F1, with the largest gap against Lookback Lens. The paper takes this to indicate that ContraTales is specifically challenging for methods that rely on new entities, lexical novelty, or simple attention heuristics, and that the residual probe is capturing deeper logical consistency.

The attribution analysis uses gradient-times-activation. For the probe score

$(x_0,\dots,x_{T-1})$2

the gradient with respect to the residual stream is

$(x_0,\dots,x_{T-1})$3

For a module output $(x_0,\dots,x_{T-1})$4, including MLP outputs $(x_0,\dots,x_{T-1})$5 or attention-head outputs, the contribution is defined as

$(x_0,\dots,x_{T-1})$6

Averaging over tokens in the final sentence and then across top-$(x_0,\dots,x_{T-1})$7 hallucination examples yields the dataset-level attribution profile. On ContraTales, for a probe trained on layer 10 of Gemma-2-9B, MLP attributions show a sparse, consistent pattern: layer 7 contributes positively, layer 8 strongly positively, and layer 9 strongly negatively. The same late MLP sub-circuit pattern appears across CNN/DM, XSum, and ContraTales, suggesting cross-domain stability in how contextual inconsistency is represented.

The domain adaptation results are similarly pronounced. After finetuning observer models for two epochs on 1,000 correct-only ContraTales stories using standard supervised finetuning language-model hyperparameters, and then retraining the probe, F1 improves by +0.10 for GPT-2-small, +0.17 for Gemma-2-2B, and from 0.75 to 0.89 for Gemma-2-9B, a gain of +0.14. This suggests that additional exposure to in-domain correct text sharpens a preexisting latent separation between faithful and contradictory endings rather than creating an entirely new signal.

6. Design rationale, limitations, and comparative position

The benchmark’s design goals are explicit. It seeks pure logical contradictions, extremely clear ground-truth labels, and weak performance for heuristics based on lexical novelty or named entities. The synthetic setup is therefore not incidental; it is used to produce a stress test in which the hallucination is subtle and plausible, the contradictory ending is tonally similar to the correct one, and surface statistics are intentionally non-discriminative. The authors describe the benchmark as tailored to stress-test contextual hallucination detectors (O'Neill et al., 31 Jul 2025).

The same design choices also define the benchmark’s limitations. ContraTales is synthetic text generated by LLMs, so some regularities may be artifacts of the generation models. Generalization to purely human-written contradictions or to in-the-wild hallucinations is not guaranteed. The benchmark focuses on a highly structured inconsistency pattern, namely a single constraint followed by a single violating sentence. It does not cover multi-step reasoning errors, subtler pragmatic inconsistencies, or extrinsic hallucinations involving unsupported but non-contradictory claims. The paper notes that broader validation on more organic hallucinations would be valuable.

In comparative terms, ContraTales is positioned against news summarization benchmarks such as CNN/DailyMail and XSum, where hallucinations often involve fabricated details that can be detected through low n-gram overlap or new entities. ContraTales instead emphasizes logical contradictions with high lexical reuse. The paper does not exhaustively compare it against named benchmarks such as TruthfulQA or HaluEval, but it frames ContraTales as complementary to heuristic-friendly synthetic benchmarks by targeting a different failure mode: contextual logical inconsistency.

A common misconception would be to treat the benchmark as a general measure of hallucination in all forms. The paper’s construction does not support that broader interpretation. A more precise reading is that ContraTales isolates one particularly clean and difficult subproblem—logical violation of local textual constraints—and is valuable exactly because of that restriction.

7. Release, usage, and broader significance

The paper states that it releases the 2000-example CONTRATALES benchmark. The excerpt does not provide a URL or license text, but the resource is intended to be public. As implied by the description, each example includes a story prefix, a correct concluding sentence, and a contradictory concluding sentence, from which researchers can derive labeled pairs such as prefix plus correct ending and prefix plus contradictory ending. No explicit train/validation/test split is specified in the excerpt; the study primarily uses 5-fold cross-validation, and the same regime can be adopted by subsequent work (O'Neill et al., 31 Jul 2025).

The benchmark supports several research uses. It can be employed to train and evaluate hallucination detectors on a generator-agnostic basis, since the observer model sees only text. It can be used to probe internal representations by extracting hidden states or residual streams and training probes to recover hallucination labels. It can support generation-time interventions in which detection signals are used to steer or filter outputs. It is also suitable for domain adaptation studies that finetune models on only the correct endings or full correct stories and then test whether labeled detection improves.

Its broader significance in the paper is twofold. Empirically, ContraTales provides a hard, logic-focused benchmark on which a single linear residual direction achieves strong F1, outperforms surface heuristics by large margins, and appears across domains and model sizes. Mechanistically, the consistency of the late-layer MLP attribution pattern suggests that contextual contradiction may be tracked by a shared, low-dimensional internal representation. The paper’s additional finding that manipulating this direction in a generator steers hallucination rates is not specific to ContraTales alone, but within the benchmark’s setting it reinforces the interpretation that the detected signal is not merely correlational. A plausible implication is that ContraTales functions simultaneously as a dataset and as a diagnostic instrument for studying how LLMs internally encode contextual logical consistency.