Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

Abstract: When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention patterns maintaining a last-in, first-out ordering. However, it remains unclear whether these representations are causally used or merely decodable. We examine this gap in transformers trained on the Dyck language (a formal language of balanced bracket sequences), where the hierarchical ground truth is explicit. By probing and intervening on the residual stream and attention patterns, we find that depth, distance, and top-of-stack signals are all decodable, yet their causal roles diverge. Specifically, masking attention to the true top-of-stack position causes a sharp drop in long-distance accuracy, while ablating low-dimensional residual stream subspaces has comparatively little effect. These results, which extend to a templated natural language setting, suggest that even in a controlled setting where the relevant hierarchical variables are known, decodability alone does not imply causal use.

• The paper demonstrates that although hierarchical variables can be linearly decoded from transformer states, they are not necessarily used for causal computations.
• The study employs controlled experiments with linear probing and ablation techniques, revealing that only attention-based stack routing is essential for correct performance.
• Activation patching localizes the critical computation to a specific attention head, with findings extending to natural language tasks such as subject-verb agreement.

Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

Introduction and Problem Framing

The paper "Dissociating Decodability and Causal Use in Bracket-Sequence Transformers" (2604.22128) interrogates a central question in transformer interpretability: when hierarchical variables are decodable from transformer states, does this imply those variables are causally used for computation? The authors address this question via a controlled experimental setting—the Dyck language of balanced brackets—which exposes explicit ground-truth computations of stack depth, stack-top identity, and bracket-pair distance. This framework enables clear causal analysis separating decodability (variables that can be linearly recovered from internal states) from causal use (variables necessary for model behavior).

Methodology

The authors utilize 2-layer, 1-head transformer models with varying widths and Dyck language variants of increasing complexity. Through a combination of linear probing on the residual stream and direct attention pattern analysis, they evaluate the decodability of three central latent variables:

• depth (current stack depth)
• distance (distance to matching opening bracket)
• top-of-stack identity (type of most recent unclosed bracket)

Causal interventions are implemented by ablating probe-aligned subspaces in the residual stream (depth and distance) and by masking attention from closing-bracket positions to their true stack-top opener (attention knockout). Activation patching localizes the critical point in the model where stack-top information is made sufficient for correct prediction.

Main Findings

1. Early and Robust Decodability of Hierarchical Variables

Training dynamics reveal that depth, distance, and top-of-stack identity are all strongly decodable from the model's internal states early in training, even before task convergence. These variables remain decodable under distributional shift (OOD lengths and depths), most notably for top-of-stack identity, which is robust across all splits.

2. Decodability is Insufficient for Causal Relevance

Despite high decodability, ablating the probe-aligned subspaces for depth and distance in the residual stream induces negligible accuracy drops (≤0.015), even on the most challenging test splits. In contrast, ablating random directions with the same rank produces substantial performance degradation, confirming the specificity of these findings. The implication is that while depth and distance are linearly encoded, they are not used by the model for the bracket-matching task.

3. Causal Necessity of Attention-Based Top-of-Stack Routing

Analyzing attention reveals highly sparse, stack-top-aligned patterns: at each closing bracket, layer 1 attention routes the majority (∼98–99.7%) of mass to the position of the matching opener. Masking this single crucial attention edge is catastrophic, reducing long-distance accuracy by approximately 0.97, while masking random edges only incurs a 0.01–0.02 decrease. This establishes attention to top-of-stack as causally necessary, in stark contrast to the inertness of residual-stream encodings of other hierarchical variables.

4. Localization of the Causal Computation

Activation patching shows that the retrieval of stack-top identity necessary for correct prediction is completed by layer-1 attention. Recovery from corrupted runs jumps from zero to one when patching is applied at or after this computational block and at the query position, indicating that correct computation is narrowly localized—a property highly relevant for mechanistic interpretability.

5. Extension Beyond Synthetic Grammar

A templated subject-verb agreement task in natural language replicates all three principal findings: hierarchical variables are decodable but not causally necessary, a single attention route is indispensable, and the critical computation localizes to a single attention head. This indicates the dissociation between decodability and causal use generalizes beyond Dyck languages to certain controlled NL tasks.

Implications

These results have significant implications for interpretability methodology:

• Linear probes are insufficient to establish causal relevance; intervention analyses are necessary to validate mechanistic claims about model computation.
• Stack-like attention routing can be mechanistically localized and is causally critical for explicit hierarchy-manipulating tasks in small-scale transformers.
• The findings reinforce the necessity of distinguishing between global decodability and local, pathway-dependent computation in both research and practical analysis pipelines.

Limitations include the concentration on small-scale models and synthetic/controlled language data; the multiplicity of mechanisms observed across random seeds suggests caution when generalizing to larger, less constrained, or less easily interrogable networks.

Future Directions

Possible extensions of this research include:

• Analysis of larger, deeper transformers and LLMs on naturalistic corpora to assess whether similar causal patterns and dissociations persist.
• Development of sharper formal tools for measuring the alignment of probe directions and attention-based computation.
• Extension to more complex hierarchical tasks (e.g., compositional logic, nested event detection) to test the universality of stack-based attention routing.
• Further systematic evaluation of activation patching and targeted ablation for mechanistic characterization in more general settings.

Conclusion

This work provides rigorous evidence that in transformer models trained on explicit hierarchical tasks, the presence of linearly decodable structure in internal representations does not imply causal use for model predictions. Only specific, attention-based routing mechanisms—here, stack-top attention edges—are causally necessary and sufficient. These findings necessitate reevaluation of interpretability results predicated on linear decodability and call for causal intervention frameworks as the standard for mechanistic claims in neural LLMs.