LLM Reasoning Is Latent, Not the Chain of Thought

Abstract: This position paper argues that LLM reasoning should be studied as latent-state trajectory formation rather than as faithful surface chain-of-thought (CoT). This matters because claims about faithfulness, interpretability, reasoning benchmarks, and inference-time intervention all depend on what the field takes the primary object of reasoning to be. We ask what that object should be once three often-confounded factors are separated and formalize three competing hypotheses: H1, reasoning is primarily mediated by latent-state trajectories; H2, reasoning is primarily mediated by explicit surface CoT; and H0, most apparent reasoning gains are better explained by generic serial compute than by any privileged representational object. Reorganizing recent empirical, mechanistic, and survey work under this framework, and adding compute-audited worked exemplars that factorize surface traces, latent interventions, and matched budget expansions, we find that current evidence most strongly supports H1 as a default working hypothesis rather than as a task-independent verdict. We therefore make two recommendations: the field should treat latent-state dynamics as the default object of study for LLM reasoning, and it should evaluate reasoning with designs that explicitly disentangle surface traces, latent states, and serial compute.

• The paper demonstrates that latent-state trajectories are the primary mediator of LLM reasoning, outperforming explicit chain-of-thought methods in standard tasks.
• It introduces controlled experimental designs that separately manipulate latent states, surface traces, and compute to isolate causal effects.
• Empirical results reveal that targeted latent interventions improve reasoning fidelity and reliability, supporting a paradigm shift in LLM evaluation.

Latent-State Trajectory Formation as the Default Science of LLM Reasoning

Introduction

The paper "LLM Reasoning Is Latent, Not the Chain of Thought" (2604.15726) advances a methodological argument that LLM reasoning should be conceived as the formation of task-relevant latent-state trajectories, rather than being strictly equated to explicit surface chain-of-thought (CoT) traces. This distinction is highly consequential for empirical research on reasoning benchmarks, faithfulness, interpretability, and inference-time interventions. The paper formalizes three competing explanatory hypotheses: reasoning as latent-state mediation (H1), reasoning as surface-CoT mediation (H2), and reasoning as a generic product of increased serial compute (H0). Through comprehensive synthesis of empirical, mechanistic, and survey literature, as well as compute-matched exemplars, the paper adjudicates among these hypotheses and recommends a paradigm shift toward latent-state dynamics as the default scientific object for LLM reasoning.

Formalization of the Problem and Hypotheses

The authors separate three explanatory objects:

• $S$: explicit semantic content of surface CoT,
• $Z$: task-relevant latent-state trajectory in hidden activations,
• $B$: generic serial compute budget (iterative computation/search), regardless of representational form.

Once these are disentangled, the field must choose which object to treat as primary for scientific inquiry. The three hypotheses are defined as follows:

• H1 (Latent-trajectory mediation): Reasoning is primarily mediated by latent-state trajectories; surface CoT is an interface, not the default substrate.
• H2 (Surface-CoT mediation): Reasoning is fundamentally mediated by explicit natural-language traces.
• H0 (Serial-compute null): Reasoning gains are mainly explained by increased serial compute; representational form is secondary.

These hypotheses make distinct causal predictions: interventions on surface traces should be maximally effective under H2; compute-only expansions should explain gains if H0 holds; and latent-state interventions should impact reasoning independently of surface traces for H1.

Review of Evidence and Adjudication

Evidence for H2

The strongest support for H2 comes from scenarios where explicit reasoning traces are made constitutive (e.g., symbolic solvers, forced decomposition, tool-based reasoning with intermediate outputs). In these cases, interpretability and answer fidelity improve because the visible trace becomes structurally binding. However, most empirical gains from CoT prompting are confounded with increased compute depth, undermining claims for surface mediation. Causal leverage is rarely isolated to the explicit trace itself, and ordinary CoT is often incomplete or unfaithful across tasks [turpin2023unfaithful, lanham2023faithfulness].

Evidence for H0

Empirical studies demonstrate that expanded serial budget (via search, sampling, self-consistency, or adaptive solve/verify allocation) frequently improves reasoning performance regardless of representational format [li2024serial, snell2024ttc]. Task performance can sometimes be maintained even with semantically meaningless intermediate tokens, suggesting that serial depth itself is a major driver [pfau2024dot]. Nonetheless, H0 cannot account for why specific hidden states and targeted latent interventions tightly couple to reasoning outcomes.

Evidence for H1

Multiple converging lines of evidence suggest latent-state trajectories are the privileged substrate for reasoning:

• Latency and Fidelity: Propositional probes show latent world-state representations encode reasoning commitments more faithfully and earlier than surface CoT outputs [feng2024latentworld]. Hidden states can predict answer correctness before verbalization, supporting early exit without performance loss [zhang2025selfverification].
• Latent Reasoning Interventions: Continuous latent-space reasoning, geometry-aware latent steering, and targeted latent interventions (e.g., verifier-guided control) improve reasoning accuracy independently of surface CoT [hao2024continuous, he2026latentmode, kazama2026geosteer, nguyen2026atlas, li2026ipg, saunshi2025latentthoughts].

These results are diagnostic rather than definitive. They show the strongest explanatory leverage accrues to latent-state dynamics in ordinary regimes. Boundary conditions emerge: surface mediation is locally explanatory when decomposition or explicit intermediate structure is made constitutive, and compute-first accounts dominate when search or iterative budget is the main driver.

Methodological Implications

Current poorly controlled experimental designs often fail to discriminate among H1, H2, and H0 due to confounded manipulations. The paper prescribes:

• Factorized Contrasts: Designs must separately manipulate $S$, $Z$, and $B$, employing compute-matched controls. Intervention effects must be pre-registered with respect to which hypothesis they discriminate.
• Admissible Arm Structure: Minimal discriminative designs require experimental arms: baseline, targeted surface manipulation (+ control), targeted latent intervention (+ control), and compute-only expansion.
• Commitment Tracking: Readouts must specify whether answer-relevant commitment follows explicit trace $S$ or arises in latent state $Z$ independently.

The recommendations are: (1) treat latent-state dynamics as the default object of LLM reasoning; (2) employ compute-audited, factorized experimental designs to disentangle $S$, $Z$, and $Z$0.

Empirical Adjudication

A two-tier adjudication program (controlled regime matrix and naturalistic transport suite) with compute-audited comparisons finds:

• Ordinary Regimes: Latent-state policies ($Z$1) consistently outperform surface ($Z$2) and compute ($Z$3) policies in arithmetic and standard reasoning tasks (e.g., GSM8K-Platinum). Mediator tests show causal leverage for $Z$4—early prediction, necessity under ablation, sufficiency under patching, specificity versus shams, and less performance degradation than surface corruption.
• Constitutive Regimes: Surface mediation dominates when explicit intermediate structure is essential (e.g., retrieval-plan gating in HotpotQA).
• Compute-heavy Regimes: Serial compute policies yield strongest gains in search-dominant tasks (e.g., MATH).
• Mixed Regimes: No single substrate uniformly wins (e.g., execution-backed code synthesis); boundary regimes reflect regime-dependent explanatory dominance.

Regime-level verdicts empirically corroborate the theoretical argument: latent trajectory mediation is dominant in ordinary LLM reasoning regimes unless specific structural conditions favor H2 or H0.

Related Work

The position is contextualized relative to strands of research on:

• Latent reasoning and direct latent control interventions [feng2024latentworld, hao2024continuous, he2026latentmode].
• Faithfulness and interpretive privilege of surface traces, including their limitations [lanham2023faithfulness, turpin2023unfaithful, chen2025dontsay].
• Role of serial computation and test-time budget [li2024serial, snell2024ttc, singhi2025whentosolve].
• Mechanistic mediation studies (circuits analysis, iteration heads) reveal internal structure but, unless paired with matched controls and causal tests, cannot decisively adjudicate among the hypotheses [cabannes2024iteration].

Implications and Speculation

Practical Implications

For LLM evaluation, intervention, and safety, research should privilege latent-state monitoring, mediator qualification, and latent interventions rather than relying only on surface chain-of-thought traces for interpretability or control. New benchmarks and evaluation protocols should audit and factorize serial compute to disentangle reasoning success from generic iterative depth.

Theoretical Implications

This paradigm shift reorients the field toward internal trajectory analysis, stimulating further work on causal mediation, mechanistic interpretability, latent-state representation, and their correspondence to reasoning performance. The position is falsifiable: it requires revision if compute-audited experiments consistently show surface interventions or generic budget expansions surpass the causal leverage of latent-state trajectories.

Future Directions

Progress will depend on experimental designs that more cleanly distinguish latent, surface, and budget objects. Insights from mechanistic causality, attribution, and feature steering will become central to reasoning research. This latent-first stance may catalyze advances in self-verification, early commitment detection, and reliability diagnostics for next-generation LLMs.

Conclusion

The paper substantiates that latent-state dynamics—not explicit surface chain-of-thought—should be the default scientific object for LLM reasoning. The empirical winner follows a regime-based map: latent mediation is dominant in ordinary reasoning, surface mediation prevails in constitutive regimes, and serial compute is decisive where iterative search dominates. Methodologically, the strong claim is that research must treat latent trajectory formation as the working hypothesis whenever surface trace, latent trajectory, and compute can be meaningfully separated. This stance provides a disciplined foundation for future work on LLM reasoning.