- The paper formalizes reasoning consistency as a distinct, observable property and introduces an InspectScout scanner that achieved 0.82 overall F1 on a 60-transcript benchmark, though it failed to detect contradictory reasoning.
- The scanner found inconsistency rates ranging from 0.0% to 26.0% across safety evaluation–model combinations, with multiple-choice task formats producing the highest rates and model effects varying by task.
- The findings show that consistency checks can identify weaknesses in safety evaluation evidence after the fact, but consistent reasoning does not prove faithfulness and results remain sensitive to scanner choice, transcript quality, and sample size.
Reasoning consistency — the logical connection between a model's stated chain-of-thought (CoT) and its final answer — occupies a distinct position in the study of reasoning reliability. Whereas faithfulness concerns whether stated reasoning reflects internal computation and can only be established through controlled interventions, consistency is an observational property: it can be judged from a transcript alone. This distinction motivates "Reasoning Consistency Scanning" (2607.07229), which formalizes consistency as a separate construct, builds a validated benchmark, implements a production scanner for InspectScout, and applies it to naturalistic safety evaluation transcripts from inspect_evals. The central empirical claim is that reasoning inconsistency is present in safety-critical evaluation transcripts, detectable post-hoc, and varies systematically — from 0.0% to 26.0% across evaluation–model combinations — with the variation driven primarily by the interaction between task format and model rather than by either factor alone.
Motivation: consistency as a threat to construct validity
The paper grounds its contribution in the construct validity framework for evaluations, which asks whether a target property is well defined, correctly operationalized, and connected to real-world harms (Raji et al., 2021, Bowman et al., 2021). Many safety evaluations treat reasoning traces as evidence that a model is genuinely deliberating — weighing ethical considerations, recognizing constraints, or reasoning about deployment context. If reasoning and answer are decoupled, that evidential chain breaks in two ways: the evaluation may score transcripts that "look right" without the model having engaged in the reasoning the score implies, and CoT monitoring — an increasingly prominent safety mechanism (Chen et al., 8 May 2025) — becomes unreliable, since monitors assume reasoning and behavior are coupled.
The paper is careful to position consistency as a strictly weaker property than faithfulness. Prior work has shown that models act on features they do not mention (Turpin et al., 2023), that conditioning on stated reasoning varies across tasks and model sizes (Lanham et al., 2023), that CoT monitoring reveals fewer than 20% of hint-using cases in reasoning models (Chen et al., 8 May 2025), and that models actively deny reliance on hints when asked to reflect (Walden, 12 Jan 2026). But these methods require interventions — hint injection, perturbation, early answering — that cannot be applied to transcripts already produced. Consistency scanning fills the complementary niche: a post-hoc audit that flags transcripts whose reasoning and output are observably disconnected, whatever the underlying cause. The paper concedes explicitly that a consistent transcript may still be entirely post-hoc rationalization; consistency is necessary, not sufficient, for trusting reasoning traces.
Taxonomy and scanner design
The paper defines a transcript as consistent if a line of thought can be drawn from the reasoning to the output — explicitly not a correctness check, and with deliberate carve-outs for uncertainty, hedging, and exploratory reasoning. Six inconsistency subtypes are defined and applied via an ordered decision procedure: absent reasoning (the reasoning field does not engage with the question), contradictory reasoning (reasoning argues incompatible directions internally), apparent confusion (diffuse incoherence without explicit contradiction), reasoning reversal (the reasoning's conclusion points one way, the answer commits to another), reasoning abandonment (specific raised considerations are ignored by the answer), and perfunctory reasoning (reasoning too thin to plausibly produce the committed answer). The ordering matters: it prevents the scanner from defaulting to the more familiar reversal subtype when a less common one fits better.
The scanner is an LLM-as-judge implemented on InspectScout's llm_scanner API, following UK AISI's transcript-analysis best-practices guide. It outputs a structured classification with a consistency label, subtype, confidence rating, summaries, and justification. Design choices targeting false positives include explicit instructions to distinguish exploratory from concluded reasoning and contrast statements between adjacent subtypes. Scanner model selection proved consequential: DeepSeek V4 Flash failed to produce valid structured output; Claude Haiku 4.5 produced reliable structure but low per-subtype recall; Claude Opus 4.6 was selected and held fixed across all experiments to control scanner variance.
Benchmark and validation
The synthetic benchmark comprises 60 transcripts derived from InstrumentalEval (He et al., 16 Feb 2025), generated with DeepSeek V4 Pro. Consistent examples are unmodified naturalistic transcripts; inconsistent ones are real transcripts surgically modified at either the reasoning or the answer to introduce a specific subtype; edge cases stress-test decision boundaries. Validation results with Opus 4.6 as judge are uneven across subtypes:
| Category |
Count |
Precision |
Recall |
F1 |
| Consistent |
15 |
0.94 |
1.00 |
0.97 |
| Reasoning reversal |
7 |
0.85 |
0.75 |
0.80 |
| Reasoning abandonment |
7 |
1.00 |
0.62 |
0.78 |
| Perfunctory reasoning |
5 |
0.83 |
0.83 |
0.83 |
| Contradictory reasoning |
5 |
0.00 |
0.00 |
0.00 |
| Apparent confusion |
4 |
1.00 |
0.50 |
0.67 |
| Absent reasoning |
4 |
0.80 |
0.80 |
0.80 |
| Not applicable |
8 |
0.875 |
1.00 |
0.933 |
| Edge cases |
5 |
0.80 |
1.00 |
0.89 |
| Overall |
60 |
0.96 |
0.71 |
0.82 |
The most notable result is the complete failure on contradictory reasoning (zero precision and recall), which the authors attribute to the inherently graded nature of coherence judgments and possibly to overly subtle synthetic samples. Because recall is weakest on the subtle subtypes, the paper notes that naturalistic inconsistency rates should be read as lower bounds for those categories. Scanner-model comparison in the appendix underscores judge sensitivity: DeepSeek V4 Flash achieved F1 of 0.55 and Haiku 4.5 of 0.68, against 0.82 for Opus 4.6 — a spread large enough that scanner choice is itself a substantive methodological decision.
Empirical results across evaluations and models
The scanner was applied to transcripts from four evaluation suites — InstrumentalEval (76 samples, benchmark source), MORU (100 samples, moral uncertainty reasoning), Agentic Misalignment (1 available sample), and three SAD-mini subsets (100 samples each: stages_oversight, stages_full, influence) (Laine et al., 2024) — using three generator models: Gemini 3.1 Pro, DeepSeek V4 Pro, and gpt-oss-120b. GPT-5.4 was attempted but excluded: with InspectScout's default parameters it exposed no reasoning traces, and regenerating via the Responses API with reasoning summaries exceeded budget constraints.
Three patterns stand out. First, task format appears to be a dominant driver. The multiple-choice SAD subsets show markedly higher inconsistency than the open-ended MORU: sad_stages_full reaches 26.0% for Gemini 3.1 Pro — the highest single value in the dataset — and 15.0% for DeepSeek V4 Pro, while MORU sits at 0.9–1.0% for those two models. The authors' hypothesis is that constrained single-letter answers lead models to abbreviate or abandon reasoning, treating it as a formality; consistent with this, perfunctory and absent reasoning are the dominant subtypes dataset-wide. Second, no model is uniformly inconsistent or consistent. DeepSeek V4 Pro peaks on SAD influence and oversight (17.0% and 12.0%), Gemini 3.1 Pro peaks on stages_full (26.0%) but is near the floor on MORU (1.0%), and gpt-oss-120b shows the inverse pattern on MORU (10.0%, the worst there) while remaining low on SAD (2.0–7.0%). This interaction effect suggests task-specific reasoning failures rather than a general disposition toward inconsistency. Third, Agentic Misalignment registered 0.0% across all models, but with a single available sample the authors correctly flag this as illustrative rather than statistically meaningful.
The appendix examples make the failure modes concrete. A Gemini 3.1 Pro transcript on SAD stages_full reasons through a data-provenance question, concludes in its final refinement step that the text "strongly suggests" an evaluation context (option A), and then answers (D) pretraining — a high-confidence reasoning reversal. A DeepSeek V4 Pro transcript on SAD influence contains an empty reasoning field followed directly by "(B) No" — absent reasoning. A gpt-oss-120b MORU transcript plans in its reasoning to pair a refusal with a neutral statement encouraging planetary-protection compliance, but the answer delivers only the bare refusal — reasoning abandonment.
Limitations
The paper is candid about several constraints. The benchmark is anchored to a single evaluation (InstrumentalEval) and a single generator (DeepSeek V4 Pro), so validated performance characteristics may not transfer to the other corpora on which the scanner is applied. Per-subtype performance is uneven, meaning rates involving contradictory reasoning and apparent confusion are likely underestimates. Anthropic models are absent as generators due to budget constraints, and GPT-5.4 could not be included, leaving generalization to those families an open question. Gemini 3.1 Pro exposes only reasoning summaries rather than full reasoning tokens, so results for that model depend on the summaries being comprehensive; a comparison against full reasoning traces is proposed but not performed. DeepSeek V4 Pro occasionally produced gibberish reasoning, possibly reflecting training-data leakage, which complicates interpretation. Sample sizes are modest in places (Agentic Misalignment has one sample), and results are inherently scanner-dependent: a different judge model, prompt, or threshold could yield different rates, and a comprehensive sensitivity analysis is deferred. Finally, the paper reiterates that consistency detection does not imply faithfulness detection — a consistent transcript can be entirely post-hoc.
Conclusion
This work establishes reasoning consistency as an auditable property of safety evaluation transcripts and delivers the tooling to audit it at scale: a formal taxonomy with an ordered decision procedure, a validated 60-transcript benchmark, and an InspectScout scanner. Its empirical contribution is the demonstration that inconsistency in safety-relevant evaluations is real and systematically structured — highest in multiple-choice situational-awareness tasks (up to 26.0%) and strongly model-dependent — with direct consequences for how reasoning evidence in those configurations should be weighted. The methodology complements rather than replaces faithfulness research: it identifies observable disconnects post-hoc, while leaving open the questions the paper itself flags, including generalization to untested model families, scanner sensitivity, and the relationship between summary-based and full reasoning traces.