Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns

Abstract: We show that the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis. Individual dimensions encode semantic content via their signs and confidence via their magnitudes, functioning as independent binary registers. We validate this Bag of Dims framework across three model families (Qwen 3.5-4B, Gemma 3-4B, Mistral 7B) through four progressive experiments. Sign patterns alone carry predictive content: replacing all magnitudes with unity achieves 72-93% top-5 next-token accuracy through the LM head, and pure Hamming scoring without any decoder reaches 80-90% top-4096. These sign patterns organize into semantic features: using a single-token type cache (one forward pass per vocabulary token, no context), we discover 175 categories via per-dimension sign consistency (mean AUC 0.80) from 50 anchors with zero training. A trained probe adds only +0.018 AUC and converges to axis-aligned weights, confirming negligible cross-dimension structure. This structure extends to attention: all 175 categories remain discoverable in K and V projections. On the write side, static FFN weight inspection links 20% of features to individual writer neurons (>0.70 agreement; random controls: 0%), with top-200 neuron coalitions achieving >0.70 agreement on 99.9% of prototypes via majority vote. Fully unsupervised discovery (random seeds, no labels) scales to 1500 features at 100% yield and 99% sparsity across all three models, with pairwise MI of 0.0014 bits confirming low inter-dimension coupling. These results establish that the standard basis already suffices for feature reading throughout the transformer compute pathway, requiring no training, no optimization, and no GPU-days beyond a single forward pass per vocabulary token.

• The paper demonstrates that transformer hidden state dimensions inherently act as independent binary registers encoding semantic content via sign patterns, with magnitude reflecting confidence.
• It introduces a training-free bag-of-dims framework that efficiently discovers hundreds of features using raw sign patterns, achieving up to 93% top-5 next-token accuracy in experiments.
• The work implies that costly learned rotations are unnecessary, suggesting a scalable, architecture-general method for interpretable feature extraction in mid-scale transformer models.

Mechanistic Interpretability via Dimension-Level Sign Patterns in Transformers

Introduction

The paper "Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns" (2606.12629) challenges the prevailing notion that interpretability in transformer models requires costly feature extraction and non-trivial learned rotations. It posits, and empirically demonstrates, that in trained transformers, the standard basis—the native hidden state axes—already forms a feature basis wherein individual hidden state dimensions operate as functionally independent binary registers. Each dimension encodes semantic content through its sign, while magnitude represents confidence. This finding supports a framework for training-free, architecture-general interpretation of transformer representations. The study validates this through comprehensive experiments across three LLMs: Qwen 3.5-4B, Gemma 3-4B, and Mistral 7B.

Empirical Evidence for Standard Basis Feature Encoding

Content-Independence and Structural Regularity

The analysis begins by visualizing all hidden state dimensions across layers and diverse prompts. The authors observe that, irrespective of input content, the population of dimension values manifests a striking, content-invariant expanding structure, emerging only in trained models.

Figure 1: All 2560 hidden state dimensions in Qwen 3.5-4B across layers and prompts—an expanding, content-independent structure reflecting learned organization.

Comparative analysis of trained versus randomly initialized weights confirms this structure arises from training. Random weights show only uniform expansion, while trained weights display a characteristic asymmetric “diamond” geometry, consistent across architectures.

Figure 2: Trained models exhibit an organized ‘diamond’ structure, absent in random (untrained) models, indicating the emergent organization due to learning (Qwen 3.5-4B left; Gemma 3-4B right).

Zooming into trajectories of individual dimensions, the study reveals they follow statistically independent paths, with low pairwise mutual information—on the order of $0.0014$ bits at the type level. Dimensions are thus functionally independent, justifying the 'bag-of-dims' architecture.

Figure 3: Example dimension trajectories exhibit mutual independence, with consistent behavior across prompt domains.

The Bag-of-Dims Framework

Definition and Discovery Pipeline

The framework treats each hidden state as $D$ independent binary registers, where $\mathrm{sign}(h[d])$ encodes semantics and $|h[d]|$ quantifies confidence. For each semantic category, anchor tokens are selected, and per-dimension AUC is computed to discover dimensions with consistent sign patterns—these form feature prototypes. Scoring new tokens reduces to instant sign-matching against registered dims, requiring no parameter learning.

Figure 4: Discovery involves profiling anchor tokens for consistent sign patterns, building a registry. Scoring any new token is then a sign-matching operation against these registered dims.

Zero-Training, Maximal Scalability

Feature discovery is realized using a type-level cache—a one-time pass of every vocabulary token (no context) through the model. This is maximally efficient, taking ∼20 minutes per model (on a single GPU) and enabling rapid, label-free discovery of hundreds of features.

Experimental Validations

Sign Patterns as Information Carriers

A major empirical claim is that sign patterns alone confer high next-token prediction power. Removing all magnitude information and using pure sign-based similarity, the models attain 72–93% top-5 next-token accuracy (top-1: 49–73%, model-dependent). In a pure Hamming scoring regime (no learned decoder), sign context-majority matching achieves 80–90% top-4096 accuracy. Random permutations ablate all structure, confirming the informativeness of sign patterns.

Feature Discoverability Without Training

Using 50 anchor tokens per semantic category (175 total categories), prototypes constructed from per-dimension sign AUC achieve mean AUCs of 0.80–0.84 (cross-model). Trained linear probes and MLPs provide only +0.018 AUC improvement and converge to axis-aligned weights, confirming the bag-of-dims structure is functionally complete for feature extraction. This further indicates negligible cross-dimension structure exploitable by more complex probes.

Feature Persistence Through Attention

Across the full compute pathway, including attention KV projections (which involve nontrivial learned transforms), discovered features persist and remain independently discoverable, indicating that K and V matrices preserve axis-aligned structure rather than destroying it.

Axis-Aligned FFN Writing and Circuit Tracing

Weight-inspection analysis demonstrates that discovered features can be attributed to specific FFN neurons at the preceding layer. Approximately 20% of features show $>$0.70 sign agreement with single neurons’ write vectors. Coalitions of the top 200 neurons reconstruct 99.9% of prototypes with $>$0.70 agreement, indicating a distributed, axis-aligned write mechanism.

Unsupervised Discovery and Feature Independence

The unsupervised discovery protocol produces 1500 features at 100% yield and 99% sparsity, with low inter-dimension mutual information. Across all analyses, functional independence between dimensions is robustly validated.

Theoretical and Practical Implications

The ablation studies and random-initialization controls confirm that this structure is not an architectural artifact but a product of training. The findings suggest that costly optimization methods such as sparse autoencoders, or complex interpretations of linear probe directions, may be unnecessary for a large class of mechanistically meaningful features in mid-scale transformer models. Instead, per-dimension sign patterns suffice for feature extraction and interpretation, with maximal efficiency and minimal computational overhead.

Beyond practical speedups in interpretability, this result refines the mechanistic understanding of internal representation in transformers. Feature specificity is enforced within each layer's output coordinate system (i.e., layer-specific, not globally stable across layers). The combinatorial capacity afforded by overlaps between high-dimensional, sparse sign patterns may contribute significantly to the expressive power and compositionality in LLMs. This also positions the standard basis as not merely sufficient but perhaps optimal for certain classes of feature analysis.

Potential limitations remain regarding extension to very large models (e.g., 70B+ parameters), and the characterization of low-magnitude, sign-flipping dimensions in natural contexts. Fine-grained behavioral control and steering using this framework remain challenging, and broader feature coverage may still require complementary approaches.

Speculation on Future Developments

If the bag-of-dims phenomenon scales to larger models and more complex settings, it could standardize highly efficient, training-free interpretability protocols, accelerating both research and safety audit pipelines. Moreover, the ease with which axis-aligned features can be discovered suggests new approaches to representation engineering for AI steering, controllability, and transparency. This may also inform future architectures by incentivizing learned representations with maximal per-dimension independence or sparsity, optimizing both interpretability and computational efficiency.

Conclusion

The study rigorously demonstrates that transformer hidden state dimensions in the standard basis operate as independent, interpretable binary registers, encoding semantic content via their signs and confidence via magnitudes. This axis-aligned structure enables training-free, scalable, and architecture-general feature discovery, as validated across multiple architectures and experimental paradigms. Implications span both mechanistic understanding and practical interpretability of LLMs, with strong support for the sufficiency—and computational advantages—of the standard basis for a broad class of mechanistic analyses.