Valid-Answer-Invalid-Reasoning (VAIR)

• VAIR is a phenomenon where a system produces a correct final answer despite flawed, unsupported, or invalid intermediate reasoning steps.
• Across forecasting, mathematics, education, and multilingual reasoning, evaluations separate answer correctness from process validity using metrics like key-event recall and source precision.
• Mitigation strategies focus on process-sensitive evaluation and verifier training to ensure systems are 'right for the right reasons' by flagging reasoning inaccuracies.

Valid-Answer-Invalid-Reasoning (VAIR) denotes a class of failures in which a system produces a correct final answer while the supporting reasoning is invalid, unsupported, or otherwise non-diagnostic of genuine competence. Across recent work, the same phenomenon appears under closely related names: in forecasting, a correct forecast may rely on post-resolution or fabricated evidence rather than temporally valid reasoning; in mathematical reasoning, a correct solution may contain at least one incorrect intermediate step; in education, the “Correct Answer Trap” describes correct answers reached through misconceptions; in multilingual evaluation, a reasoning trace may fail to support the stated answer; and in visual reasoning, “false positive” reasoning denotes a correct answer paired with a flawed reasoning path (Chi et al., 10 Jun 2026, Xia et al., 2024, Imran et al., 22 Jun 2026, Ovalle et al., 27 Dec 2025, Zhang et al., 6 Aug 2025).

1. Conceptual scope and domain-specific terminology

VAIR separates two properties that are often conflated: outcome validity and process validity. A correct answer can arise from memorization, shortcutting, post-hoc rationalization, or logically defective inference. Conversely, a system can exhibit relatively well-grounded reasoning yet still fail at the final decision step. This decomposition recurs across domains because many evaluation protocols have historically rewarded only answer correctness.

Domain	Term used	Core criterion
Forecasting	VAIR	Correct forecast with invalid evidence and/or causal reasoning
Mathematical reasoning	VAIR	Correct final answer with at least one incorrect step
Education	Correct Answer Trap	Correct answer with flawed method or misconception
Multilingual CoT	Reasoning–answer misalignment	Trace does not support the stated answer
Visual reasoning	False positive reasoning	Correct answer with flawed reasoning path

A general formalization appears in the VAIR dataset for large reasoning models. For a math problem $P$ with premises $\Pi$ and ground-truth answer $A^*$, a solution $s = (\phi_1,\ldots,\phi_T, A)$ is VAIR when the final answer is correct but at least one step is invalid:

$$VAIR(P, s) \Leftrightarrow [FinalAnswer(s) = A^*] \wedge [\exists i, \neg ValidStep(s, i)].$$

This formulation isolates reasoning validity from answer validity and makes explicit that VAIR is not merely “bad reasoning”; it is specifically the case where bad reasoning is masked by a valid outcome (Sun et al., 31 May 2026).

The phenomenon is not restricted to formal derivations. In WorldReasoner, “invalid” includes temporally invalid or fabricated sources, irrelevant or weakly grounded sources, causal stories that fail to recover hindsight-validated key events, and correctness attributable to parametric memorization rather than genuine prospective reasoning (Chi et al., 10 Jun 2026). In educational feedback, a student can reach the correct numerical answer using a misconception rule; the answer is right, but the method signals misunderstanding rather than mastery (Imran et al., 22 Jun 2026). In multilingual reasoning, the answer may be correct while the trace supports a different option or is inconclusive, making the answer evidentially unsupported (Ovalle et al., 27 Dec 2025).

2. Formal evaluation frameworks

Recent work operationalizes VAIR by decomposing evaluation into separate axes for answers, evidence, and reasoning rather than relying on a single correctness label. WorldReasoner makes this explicit for temporally valid event forecasting. A resolved forecasting task is

$q_i = (x_i, t_i^o, t_i^{sim}, t_i^{res}, y_i),$

where $x_i$ is the question, $t_i^o$ the earliest forecastable date, $t_i^{sim}$ the simulated forecast date, $t_i^{res}$ the resolution date, and $\Pi$0 the resolved outcome. The Temporal Gateway exposes only pre-date evidence,

$\Pi$1

and the agent submits

$\Pi$2

with predicted answer $\Pi$3, probability $\Pi$4, cited sources $\Pi$5, rationale $\Pi$6, and optional forecast-time causal graph $\Pi$7. Evaluation then separates outcome quality, evidence quality, and reasoning quality:

$\Pi$8

Outcome quality is measured by accuracy, Brier score, and log score; evidence quality by source precision

$\Pi$9

and reasoning quality by key-event recall

$A^*$0

This design is explicitly aimed at exposing cases where an answer is correct but the path is invalid for forecasting (Chi et al., 10 Jun 2026).

In mathematical reasoning, ReasonEval formalizes process quality through validity and redundancy. Each step is labeled as positive, neutral, or negative. Given stepwise probabilities $A^*$1 over these classes, step validity and redundancy are

$A^*$2

At solution level, validity is aggregated by worst-step logic,

$A^*$3

while redundancy is aggregated by

$A^*$4

A natural VAIR rule is then: final answer correct and $A^*$5. The paper uses $A^*$6 for general automatic labeling and $A^*$7 for high-recall VAIR estimation on MATH (Xia et al., 2024).

Other frameworks instantiate the same separation in different ways. In multilingual evaluation, reasoning–answer support is defined by an evaluator $A^*$8 that infers which option the trace supports once the final answer string has been removed. With correctness $A^*$9 and support $s = (\phi_1,\ldots,\phi_T, A)$0, VAIR is

$s = (\phi_1,\ldots,\phi_T, A)$1

and overall misalignment is measured by the Trace Inconsistency Rate,

$s = (\phi_1,\ldots,\phi_T, A)$2

Conditional VAIR among correct answers is reported as $s = (\phi_1,\ldots,\phi_T, A)$3 (Ovalle et al., 27 Dec 2025).

Trace-collapse work uses a structural decomposition. If an output is $s = (\phi_1,\ldots,\phi_T, A)$4, structural validity is split into valid reasoning (VR), empty reasoning (ER), missing reasoning (MR), and truncated reasoning (TR). With overall accuracy $s = (\phi_1,\ldots,\phi_T, A)$5 and reasoning-conditioned accuracy $s = (\phi_1,\ldots,\phi_T, A)$6, the VAIR rate is

$s = (\phi_1,\ldots,\phi_T, A)$7

This equation makes visible the portion of correct answers that arrive with invalid explicit traces (Twist et al., 20 May 2026).

Verifier work also exposes VAIR by separating answer accuracy from rationale quality. REPS evaluates Rationale Accuracy (RA), which measures whether a verifier selects the candidate with a valid rationale, from Answer Accuracy (AA), which counts any candidate with the correct final answer as acceptable (Kawabata et al., 2024). This distinction matters because answer-only supervision can systematically treat VAIR cases as positive training examples.

3. Empirical manifestations across domains

WorldReasoner provides one of the clearest large-scale demonstrations. The benchmark contains 345 resolved tasks derived from 14,141 articles, with graphs covering 8,087 extracted events. Weighted across contamination-filtered pairs, outcome accuracy is 58.7% for Vanilla, 56.6% for Causal Simulation, 68.8% for Search-Enabled, 64.4% for SE Graph, 74.7% for Near-Resolution, and 88.8% for Real-Time. Temporally valid retrieval is therefore the strongest driver of accuracy, while graph construction improves key-event recovery but not accuracy by itself. In SE Graph, correct forecasts have higher KER and SrcP than incorrect forecasts—10.8% versus 8.0% for KER, and 62.6% versus 48.0% for SrcP—yet even correct graph-enabled forecasts recover only about 11% of key events on average and still cite many sources outside the hindsight-evidence set. The paper also gives a canonical contamination example: in Vanilla, GPT-5.4 drops from 69.2% unfiltered accuracy to 52.7% after cutoff filtering (Chi et al., 10 Jun 2026).

In mathematics, ReasonEval shows that final-answer accuracy does not guarantee better reasoning quality. On MR-MATH invalid detection, ReasonEvalLlemma-34B reaches F1 79.6 at solution level and 77.5 at step level, outperforming GPT-4 prompting and Math-Shepherd baselines. More directly relevant to VAIR prevalence, on MATH the estimated false positive rate among correct answers remains substantial: for LLaMA2-13B-PRM800K, accuracy 7.4% corresponds to 40.6% FPR; for WizardMath7B-V1.1, accuracy 31.0% corresponds to 16.7% FPR; and stronger models often plateau around roughly 20% false-positive rates among correct answers (Xia et al., 2024).

Verifier training results in StrategyQA make the same point in a different form. Although 59% of generated solutions contain the correct answer, only 19% of those correct-answer solutions have valid rationales. Training verifiers on valid rationales rather than correctness-only labels markedly improves Rationale Accuracy across ARC-Challenge, DROP, and StrategyQA (Kawabata et al., 2024).

Educational data highlight a different statistical regime: VAIR is rare but operationally costly. On 20,964 real student responses, True Misconception—correct answer with flawed reasoning visible in the explanation—accounts for 1.6% overall. In the test set, there are 61 TM cases and 2,221 True Correct cases, so prevalence among correct answers is about 2.7%. Gemma 4 26B detects about 83.6% of TM cases, but with an 18.0% false-positive rate on TC; at realistic prevalence, precision is only about 11%, so false alarms outnumber genuine detections by roughly 8 to 1 (Imran et al., 22 Jun 2026).

Cross-lingual evaluation reveals that high task accuracy can coexist with systematic reasoning–answer misalignment. Across six frontier models and six languages, non-Latin scripts show at least twice the misalignment of Latin scripts. Conditional on correct answers, English and Spanish range roughly from 0.57% to 2.37% VAIR, while Korean ranges from 6.28% to 8.91%. The paper reports, for example, that Qwen2.5-32B-Instruct in Korean attains 77.01% accuracy while exhibiting 8.88% VAIR and 13.30% overall TIR (Ovalle et al., 27 Dec 2025).

VAIR also appears in prompting studies. On all 23 BIG-Bench Hard tasks evaluated with Codex, logically invalid chain-of-thought prompts achieve performance gains similar to logically valid prompts and substantially outperform answer-only prompting. The same paper documents logical mistakes in some previously used “valid” BBH exemplars, including unjustified substitutions in multistep arithmetic and incorrect navigation coordinates, yet CoT still produced gains (Schaeffer et al., 2023).

Large reasoning models exhibit an additional asymmetry: they are often better at producing solutions than evaluating whether a correct answer was reached by invalid reasoning. On the VAIR dataset of 1,001 math problem–solution pairs with correct final answers but trivial flaws, frontier models achieve production accuracy of at least 94.7% and control-set evaluation accuracy of at least 91.9% on VAVR and at least 95.8% on IAIR, but VAIR evaluation accuracy falls as low as 47.9% for GPT 5.4 and 52.5% for GPT 5 (Sun et al., 31 May 2026).

4. Mechanisms and causal explanations

Several mechanisms recur across the literature. In forecasting, temporal leakage and parametric memorization are central. Static resolved benchmarks can be contaminated by post-date knowledge, so answer accuracy may reflect memorized facts rather than genuine prospective inference. WorldReasoner addresses this with a simulated forecast date, a Temporal Gateway, and contamination filtering, and its empirical drops under filtering show that knowledge-only systems can appear to forecast when they are in fact recalling post-start facts (Chi et al., 10 Jun 2026).

In large reasoning models, a prominent mechanism is answer confirmation bias. Chain-of-thought analyses on the VAIR dataset distinguish Independent Solving from Step Tracing and Blind Endorsement from Forced Rationalization and Strict Rejection. On VAIR examples, models frequently re-solve the problem, confirm that the final answer is correct, and then rationalize flawed steps rather than checking inferential dependencies. Linear probes show that valid versus invalid reasoning is linearly decodable on concordant cases, but below chance on fooled VAIR cases; dynamic probes show Group B traces progressively converge toward “valid” representations as the model approaches its verdict. Causal patching of answer-token representations flips both verdicts and internal trajectories, demonstrating that answer validity causally drives the mis-evaluation (Sun et al., 31 May 2026).

A related mechanistic account treats final answers as jointly influenced by two competing pathways: deliberate reasoning and direct retrieval from internal memory. Controlled perturbations show non-zero reasoning-perturbation success rates and retrieval-perturbation success rates across models and datasets, confirming that both pathways operate simultaneously. Distillation-based models exhibit higher post-hoc explanation rates and stronger retrieval dominance, whereas RL-trained models are more reasoning-dominant and more robust. This suggests that some VAIR cases arise when retrieval produces the correct answer and the model fabricates a compatible rationale afterward (Wang et al., 29 Sep 2025).

The broader notion of irrationality formalized as rational value risk reaches a similar conclusion from a decision-theoretic angle. With fixed value function $s = (\phi_1,\ldots,\phi_T, A)$8, rational value risk measures the gap between the expected utility of a rational counterpart and that of the deployed reasoning strategy. Positive RVR across GSM8K, MATH, HumanEval, UltraFeedback, AlpacaEval, and MathArena indicates that alignment can reduce but not eliminate inference-time irrationality. In this picture, VAIR is one visible manifestation of a broader gap between aligned value functions and the actual inference-time strategy (Qian et al., 26 May 2026).

Prompting studies add a more minimal explanation: logical validity may not be the active ingredient in CoT gains. On BBH, invalid CoT remains effective even without optimization, suggesting that covariates such as step decomposition, format priming, few-shot pattern matching, rationale length, and topical relevance can improve performance independently of logically valid derivation (Schaeffer et al., 2023).

5. Detection and mitigation strategies

A consistent recommendation is to evaluate along multiple axes. In forecasting, this means outcome quality, evidence quality, and reasoning quality, with explicit temporal filtering, source auditing, and hindsight graph comparison. Correctness-conditioned splits are particularly diagnostic: large gaps between correct and incorrect cases indicate that correctness is tied to grounding, while small gaps expose pockets of VAIR (Chi et al., 10 Jun 2026).

For mathematical reasoning, step-level validity signals are crucial. ReasonEval uses three-way step classification and worst-step aggregation to detect hidden invalid steps, and it can also support data selection. Filtering training traces with $s = (\phi_1,\ldots,\phi_T, A)$9 and $$VAIR(P, s) \Leftrightarrow [FinalAnswer(s) = A^*] \wedge [\exists i, \neg ValidStep(s, i)].$$0 improves validity and reduces redundancy without sacrificing answer accuracy on MMIQC’s MATH subset (Xia et al., 2024).

Verifier training benefits from rationale-aware supervision. REPS generates multiple correct-answer candidates, applies pairwise self-evaluation to choose more factually grounded and logically coherent rationales, and then trains the verifier on those selected positives. This improves Rationale Accuracy from 38.90% to 53.05% on ARC-Challenge, from 36.02% to 40.90% on DROP, and from 30.13% to 38.96% on StrategyQA while leaving task performance stable or slightly improved (Kawabata et al., 2024).

Educational deployment requires base-rate-aware triage rather than direct escalation. The proposed detect-verify-escalate pipeline distinguishes “Clear reasoning,” “Needs clarification,” “Misconception detected,” and “Wrong answer,” then routes ambiguous correct-answer cases to low-cost diagnostic follow-up questions rather than directly to teachers. This design explicitly treats VAIR detection as a routing signal rather than a final decision (Imran et al., 22 Jun 2026).

When explicit reasoning traces collapse during fine-tuning, structural mitigation is possible even without teacher traces. Masked-think and response-only loss masking prevent the model from being rewarded for empty traces. For example, in Chemistry, Qwen3-8B under standard SFT with empty-think reaches VR = 0.0% and pass@1 = 56.6%, implying VAIR = 56.6%; masked-think restores VR to 82.0% and reduces VAIR to approximately 0.0–0.1% (Twist et al., 20 May 2026).

Mechanistic interventions also exist. FARL alternates GRPO with Negative Preference Optimization–based memory unlearning to suppress retrieval shortcuts. On R1-Llama-8B, FARL reduces R-PSR and T-PSR below standard RL while improving in-domain and out-of-domain accuracy; on R1-Qwen-7B, the paper reports a 60.4% reduction in overall perturbation rate (Wang et al., 29 Sep 2025). A different line of work proposes training-free verification from internal activations: spectral diagnostics of attention graphs separate valid and invalid mathematical proofs with single-threshold accuracies of 85.9–94.9%, offering a way to flag correct answers whose internal computation lacks the spectral signature associated with coherent proofs (Noël, 2 Jan 2026).

6. Limitations and open questions

Current VAIR evaluation remains domain- and framework-dependent. WorldReasoner notes that reference graphs are not perfect causal ground truth, inter-rater reliability is not reported, and absolute KER values are conservative because of strict 7-day matching and top-5 key-event selection (Chi et al., 10 Jun 2026). In mathematical process evaluation, redundancy remains intrinsically ambiguous, thresholds affect prevalence estimates, and performance depends strongly on the backbone model and the quality of human-labeled training data (Xia et al., 2024).

Low-prevalence regimes create a separate operational challenge. In educational settings, even a strong detector produces low precision because the base rate of hidden misconceptions among correct answers is about 2.7%; this makes standalone binary flagging impractical and pushes system design toward verification and triage (Imran et al., 22 Jun 2026). Multilingual evaluation introduces further complications from translation quality, script effects, and evaluator reliability; the automated evaluator reaches only moderate agreement with human consensus, and translation artifacts remain a possible confound (Ovalle et al., 27 Dec 2025).

Some frameworks measure structural validity without directly measuring truthfulness or faithfulness. Trace-collapse metrics distinguish valid, empty, missing, and truncated reasoning, but “valid” there means parseable and complete, not necessarily logically sound (Twist et al., 20 May 2026). PDS in R2PE is based on inter-chain consistency rather than single-chain proof validity, so uniformly spurious but mutually consistent rationales may evade detection (Xu et al., 2024). Likewise, rational value risk is defined over outcome utility rather than process validity, so it captures inference-time irrationality without itself resolving whether the chain of thought is faithful (Qian et al., 26 May 2026).

The accumulated evidence nevertheless converges on a stable conclusion. Accuracy alone is insufficient wherever reasoning traces, cited evidence, or causal explanations are meant to carry epistemic weight. VAIR is not an edge case confined to one benchmark or modality; it appears in forecasting, mathematics, education, multilingual reasoning, visual reasoning, answer verification, and reasoning-model evaluation. The central methodological consequence is that systems intended to be “right for the right reasons” require explicit process-sensitive evaluation, and in many cases process-sensitive training, rather than post hoc trust in correct final answers.