What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers

Abstract: When do transformers commit to a decision, and what prevents them from correcting it? We introduce \textbf{prolepsis}: a transformer commits early, task-specific attention heads sustain the commitment, and no layer corrects it. Replicating \citeauthor{lindsey2025biology}'s (\citeyear{lindsey2025biology}) planning-site finding on open models (Gemma~2 2B, Llama~3.2 1B), we ask five questions. (Q1)~Planning is invisible to six residual-stream methods; CLTs are necessary. (Q2)~The planning-site spike replicates with identical geometry. (Q3)~Specific attention heads route the decision to the output, filling a gap flagged as invisible to attribution graphs. (Q4)~Search requires ${\leq}16$ layers; commitment requires more. (Q5)~Factual recall shows the same motif at a different network depth, with zero overlap between recurring planning heads and the factual top-10. Prolepsis is architectural: the template is shared, the routing substrates differ. All experiments run on a single consumer GPU (16\,GB VRAM).

• The paper demonstrates that small transformers exhibit early irrevocable commitment (prolepsis) using specialized attention heads to route decisions.
• The study uses cross-layer transcoders and sparse dictionary learning to expose planning circuits that standard residual methods fail to reveal.
• Interventions reveal that model layer depth, not parameter count, is key to sustaining planning decisions without later correction.

Early Irrevocable Commitment in Small Transformers: Architectural Insights into Prolepsis

Introduction and Context

The paper "What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers" (2604.15010) presents an in-depth mechanistic investigation of early irrevocable decisions—termed prolepsis—in small transformer LLMs. The authors formalize and systematically analyze the motif where, at specific internal positions, a transformer effectively commits to an output, routing this decision through specialized attention heads, which then propagate it unaltered to the model output, with no subsequent correction possible, even when the decision is erroneous.

This analysis is situated within extensive prior work on feature decomposition, activation steering, and mechanistic interpretability, but it distinctly demonstrates that standard residual stream approaches fail to reveal the underlying circuit for planning, while cross-layer transcoders (CLTs) are essential for making the phenomenon tractable. Empirically, the methodology spans open-weight small models—Gemma 2 2B and Llama 3.2 1B—analyzing tasks covering rhyme planning and factual recall.

Observability of Planning: Sparse Features Versus Residual Stream

A key assertion is that planning in transformers does not manifest as simple directions in the residual stream, in stark contrast to factual recall. The authors perform a comprehensive evaluation of six standard residual stream-based methods—including activation addition, causal activation patching, and logit lens analysis—finding 0% steering success in planning tasks. These methods, which reliably redirect factual recall (e.g., switching "Paris" $\rightarrow$ "Berlin" at almost 40% probability), yield negligible effect in planning (rhyme structure) contexts.

Instead, only sparsely activated, interpretable features extracted via CLTs—extended dictionary-learning-based methods—are sufficient to reveal the circuit underlying planning. Without the decomposition provided by CLTs, intervention protocols fail. This establishes a clear computational substrate distinction: planning is encoded outside the typical residual stream manifold, tightly localized in high-resolution, sparse activation spaces.

Localization and Geometry of Commitment

Replicating the signature "planning site" spike observed in proprietary large models [Lindsey et al., 2025], the authors conduct a detailed suppress-plus-inject intervention sweep with CLTs on open small models. Suppressing natural rhyme group features and injecting those of an alternative group at each token position, they measure the probability that the injected word is produced. The probability remains at a noise floor ($\sim10^{-8}$) across all positions, except for a sharp and exclusive spike at one "planning site" position where the committed decision is made.

Figure 1: Position sweep on Gemma 2 2B (426K CLT): single-layer spike in $P(\text{“around”})$ at the planning site, confirming positional localization of proleptic commitment.

In Gemma 2 2B, the best-case shift raises the output probability of an alternative rhyme word from $4.5\times10^{-8}$ to 0.483—a ten-million-fold increase. This sharp localization is recapitulated across Gemma and Llama variants, at both rhyme-group and word-level feature resolutions. Importantly, the commitment geometry is conserved; the planning site is the unique locus where intervention can redirect output.

Propagation: Specialized Attention Routing Heads

The paper investigates how early decisions are sustained and routed to the output layer. Standard attribution methods fail to reveal this mechanism, but by measuring attention deltas under suppress-plus-inject interventions, the authors identify a small set of functionally specialized attention heads—most notably, Gemma's L21:H5—that control the transmission of planning-site decisions to the output positions.

Figure 2: Dominant attention routing heads under suppress-plus-inject interventions; individual heads (e.g., L21:H5) channel planning site decisions, while other heads disengage (red). In Llama, dominant heads are prompt-specific.

This mechanism presents as a strong push-pull redistribution in attention. Notably, the authors demonstrate that without simultaneous suppression of the natural group, injections are $13\times$ less effective, emphasizing that attribution of functionality to heads is only visible under a dual (suppress-plus-inject) protocol. Moreover, the attractor regime for planning is "soft": output changes are linearly proportional to intervention strength, with no hard threshold. In contrast, factual recall interventions exhibit hard attractor basins and are rapidly damped.

Figure 3: Strength sweep: Linear, soft attractor behavior in planning (no threshold), with distributed attention routing in Llama versus highly concentrated routing in Gemma via L21:H5.

Architectural Depth and Commitment

The ability to sustain and irrevocably route a planning decision is not determined by parameter count, but by model layer depth. Llama 3.2 1B, with 16 layers, supports initial feature search (as evidenced by relevant CLT activations and intermediate logit lens hits), but cannot sustain a commitment: candidates dissipate before reaching the output. Gemma 2 2B, with 26 layers, channels decisions through downstream heads, supporting full prolepsis. Causal interventions removing late layers in Gemma immediately destroy rhyme planning, confirming a commitment threshold at $>$16 layers.

Prolepsis Beyond Planning: Factual Recall Circuits

Extending the analysis to factual recall, the authors find the same motif—early commitment, attention-head routing, and zero correction in later layers—even though the specific heads and network depths differ. Mid-layer heads implement planning routing; late-layer heads route factual recall. Critically, there is zero overlap between recurring planning-heads and factual recall heads at the aggregate level, establishing that prolepsis is a motif instantiated by the architecture, not specific wiring.

Implications and Prospects

These findings have several substantial theoretical and practical implications:

• Methodological Scope: Only feature decomposition methods (e.g., CLT, sparse dictionary learning) can reliably expose and intervene on planning-site decisions. Mainstream interpretability tools operating on the residual stream are blind to such motifs, suggesting a need for broader adoption of sparse techniques and more exhaustive feature dictionaries.
• Task and Domain Specificity: The visibility of prolepsis is bounded by the presence of high-frequency, domain-constraining features in training data (e.g., rhyme groups induced by web poetry). In other domains (e.g., code, cooking), exhaustive scans failed to produce analogous features, indicating limits on the tractable generalization of planning-site localization.
• Safety and Control: The observation that proleptic commitment is irrevocable—even for erroneous decisions—has implications for model steering and safety. Models that commit early are harder to redirect post-commitment, which could impact controllability.
• Circuit Divergence: The mechanistic dissociation between MLP-centric factual recall and attention-based planning circuits suggests that complex tasks in transformers may leverage orthogonal computational substrates under a shared architectural template.

Conclusion

The paper establishes with empirical rigor that small transformer architectures instantiate a robust prolepsis motif: early irrevocable commitment routed by dedicated attention heads, visible only via sparse, high-resolution feature decomposition, and corroborated across independent tasks. The minimum architectural requirement for such behavior is determined by model layer depth rather than parameter count. The prolepsis template is shared across computational regimes—planning and factual recall—but the substrate circuits (i.e., recruited heads and depth) are task-dependent and non-overlapping. These findings both refine mechanistic claims about how transformers generalize and amplify the methodological imperative to extend beyond residual stream-based interpretability.

---