Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions

Abstract: We study planning site formation in LLMs -- where internal representations of structurally-constrained future tokens form during the forward pass, and whether they causally drive generation. Using rhyming-couplet completion as a clean test of forward-looking constraint, we apply two lightweight methods (linear probing and activation patching) across Qwen3, Gemma-3, and Llama-3 at more than ten scales. Probing shows that future-rhyme information is linearly decodable at the line boundary, with signal that strengthens with scale in all three families. Activation patching reveals that only Gemma-3-27B causally relies on this encoding, exhibiting a handoff in which the causal driver migrates from the rhyme word to the line boundary around layer 30. Every other model we test conditions on the rhyme word throughout generation, with near-zero causal effect at the line boundary despite strong probe signal. We localize the Gemma-3-27B handoff to five attention heads through two-stage path patching that recover ~90% of the rhyme-routing capacity at the newline.

• The paper demonstrates that latent planning representations are detectable in LLMs via linear probing and activation patching.
• It shows that only Gemma-3-27B exhibits a causal handoff from the rhyme token to the line boundary, a phenomenon dependent on model scale and architecture.
• Mechanistic interventions pinpoint a sparse group of attention heads mediating the planning effect, highlighting distinctions between predictive and causal representations.

Locating Latent Planning in LLMs with Lightweight Mechanistic Interventions

Introduction

This paper addresses the existence and mechanistic localization of latent planning within LLMs, specifically in the context of generating structurally-constrained text such as rhyming couplets. The central focus is on identifying both where in a model's architecture planning-relevant information is represented (the "planning site") and whether such information causally drives generation decisions. The analysis compares three open-source model families (Qwen3, Gemma-3, and Llama-3) across ten model scales, leveraging the lightweight methods of linear probing and activation patching to probe and perturb internal representations. A central, novel claim is that although planning-compatible representations are linearly accessible across multiple architectures, only a specific model and scale (Gemma-3-27B) demonstrates a distinct, causally active planning site handoff from the rhyme token to the line boundary—a phenomenon strongly tied to model scale and architecture rather than simply the presence of linearly decodable information.

Experimental Setup and Notation

The core experimental scaffold involves using rhyming couplet generation as a testbed for investigating latent planning. For each model, activations are recorded at strategic token positions: the last word of the first line (rhyme-defining token) and the subsequent line boundary (newline token). Probing and patching are then performed at various layer-depths ($\ell$) and token positions ($i$). Planning-compatible representations are defined as those which enable accurate linear decoding of a future constraint (e.g. an upcoming rhyme), while a causally active planning site is one where direct intervention alters downstream model behavior.

Probing for Planning-Compatible Representations

Linear probes were trained to predict the target rhyme token from hidden activations at designated positions and depths, both on general text (The Pile) for control and on synthetic rhyming couplets. Results on general text confirm that forward-looking, task-specific information is not generically present; probe accuracy for future tokens degrades monotonically with lookahead ($k$), converging to the unigram baseline for large $k$ (Figure 1).

Figure 1: Top-5 accuracy of linear probes predicting $k$ tokens ahead in general text (Pile); accuracy decays with $k$ and meets the unigram baseline near $k=8$.

When applying probes to rhyming couplets, a sharply differentiated pattern emerges: probe accuracy for predicting the target rhyme ($r_2$) is significantly elevated at the last word and at the newline, with performance at both positions exceeding that of tokens beyond these boundaries (Figure 2).

Figure 2: Top-5 and rhyme accuracy of linear probes in rhyming couplet tasks; information is selectively concentrated at the last word and newline positions at certain layers.

The magnitude and prominence of the newline probe encoding grows with model scale, particularly in the Gemma-3 family. Quantitatively, the gap in probe accuracy between the newline and post-newline positions increases with model size, especially within Gemma-3 and the largest Qwen3 and Llama-3 models (Figure 3).

Figure 3: Maximum newline/first-generated probe accuracy gap across model size, showing clear emergence only at high scale, notably in Gemma-3 models.

These findings demonstrate that planning-compatible representations at the line boundary are emergent properties in large models, but they do not establish functional or causal involvement in generation.

Patching and Causal Assessment of Planning Sites

To discern whether the encoding sites detected by probing are causally active, activation patching was performed: hidden states from a corrupted (rhyme-shifted) prompt are substituted into the forward pass of a clean completion, and the rate at which completion is redirected towards the corrupt rhyme is measured. With this method, the Gemma-3-27B model reveals a clear handoff of causal influence from the last word to the newline position in higher layers (around layer 30), with the corrupt rhyme rate at the newline peaking at 0.63 (95% CI [0.48, 0.78]) at layer 33, while last word efficacy drops.

In contrast, Qwen3-32B and Llama-3-70B exhibit no such handoff: last-word patching remains the only robust causal driver across all layers, with newline patching remaining at noise (CI upper bounds ≤ 0.21) (see Figure 4 for Qwen3-32B).

Figure 4: Patching results for Qwen3-32B show high efficacy for the last word token at all layers but negligible effect at the newline.

This dissociation demonstrates that high probe accuracy alone is not sufficient for causal relevance; strong decodable representations may exist at the newline, but the model may not utilize them for planning unless a handoff is mechanistically realized.

Mechanistic Localization via Head Patching

To pinpoint the architectural locus of the observed handoff in Gemma-3-27B, the study performs fine-grained attention head analysis. By ranking heads by their attention weight from the newline to the last word, the authors localize the planning handoff to a sparse set of five specific heads in layers 28 and 30 (Figure 5).

Figure 5: (a) Attention weights identify a small cluster of heads (red stars) highly focused on routing the rhyme constraint. (b) Patching these top-$k$ heads at the newline recovers most of the causal effect on rhyme selection.

Two-stage path patching—where the effect of routing a corrupt residual from the last word to the newline is isolated via selected heads—confirms that most of the handoff's causal influence is mediated specifically by these few attention heads, with up to 90% of the full-residual handoff capacity recoverable by intervening on just five heads (Figure 6).

Figure 6: Two-stage path patching in Gemma-3-27B shows maximal rhyme redirection at $k=5$ heads; adding more heads degrades the effect.

This mechanics directly supports the claim that the handoff is a structured, sparsely-implemented computation, rather than a diffuse, distributed effect.

Implications, Limitations, and Future Directions

These results clarify several theoretical and practical aspects of latent planning in LLMs:

• Dissociation of encoding and use: High probe accuracy for future constraints is not a reliable indicator of functional usage in generation. Only direct interventions can establish causal roles.
• Emergent, architecture- and scale-dependent handoff: The representational handoff to the newline is not universal but arises only under specific conditions of scale and (presumably) training or architectural inductive bias, as currently only observed in Gemma-3-27B.
• Sparsity of the circuit: When present, causal planning capacity resides in a small, well-localized subset of attention heads, aligning with broader sparsity observations in mechanistic interpretability.

Future work could profitably explore several axes:

• Generalization across tasks: Extending these analyses to longer-range planning tasks, prose generation, or code completion to test the generality of planning handoff phenomena.
• Broader model families: Including additional architectures and training regimens may reveal more about the necessary (and sufficient) conditions for planning site formation.
• Causal intervention techniques: Incorporating causal scrubbing, mechanistic circuit tracing, or more targeted activation steering could further dissect the fine structure of planning circuits and address whether latent representations are merely accessible or actively read out.
• Safety assessment: Given the gap between externally observable reasoning and latent computation, these methods may be critical for robust safety evaluations and adversarial robustness research in LLMs.

Conclusion

This paper establishes a precise, mechanistic framework for detecting and localizing latent planning in LLMs. By combining linear probing with activation and path patching, it demonstrates that linearly decodable planning representations at the line boundary are common but rarely used causally; instead, most models condition on the last word throughout, regardless of scale. Only in specific architectures and scales—namely, Gemma-3-27B—does a causal representational handoff emerge, mediated by a sparse set of attention heads. These results caution against interpreting probe accuracy as direct evidence of planning behavior, reveal important distinctions in circuit implementation, and provide a scalable toolkit for future mechanistic interpretability research.

---

Reference: "Where's the Plan? Locating Latent Planning in LLMs with Lightweight Mechanistic Interventions" (2605.07984).