Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Abstract: Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through \emph{feature stability}: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

• The paper introduces a cosine similarity–based stability metric to distinguish reproducible stable features from seed-dependent unstable ones in sparse autoencoders.
• It demonstrates that stable features capture most reconstruction variance and predictive signal, while unstable features reflect rare, brittle surface triggers.
• Pooling features across seeds produces SAE dictionaries that combine high stability with near-maximal reconstruction fidelity, challenging common tradeoffs.

Seed Dependence and Feature Stability in Sparse Autoencoders

Introduction and Motivation

Sparse autoencoders (SAEs) are extensively used in mechanistic interpretability for large neural models, where they serve to decompose activations into sparse, interpretable features. A critical open question is whether these features are reproducible across independent training runs – that is, whether a given SAE feature persists when the same SAE objective is reinitialized with different random seeds. This paper rigorously interrogates this problem, introducing a per-feature stability metric: the cross-seed reappearance probability assessed via cosine similarity thresholding. The analysis spans a wide gamut of configurations (model, layer, SAE variant, sparsity budget), revealing that SAE dictionaries exhibit a pronounced division: some features are highly stable and consistently recur, while others are unstable and rarely reappear.

Figure 1: Empirical distribution of feature reappearance probabilities across training seeds, showing a substantial clustering at endpoints (stable vs. unstable).

Functional Distinction: Stable vs. Unstable Features

Stable features are those with high cross-seed recurrence probability; unstable features are rarely matched across seeds. The empirical findings demonstrate strong functional asymmetry between these sets:

• Stable features capture the bulk of reconstruction variance and prediction-relevant signal. Their activations are not only frequent but also manifest larger high-magnitude tails, indicative of robust encoding of core structural or semantic content.
• Unstable features are biased toward rare surface-form triggers: they activate on low-frequency lexical substrings, punctuation, or brittle token patterns. Their contribution to both reconstruction explained variance and downstream next-token loss is minimal.

  Figure 2: Token entropy analysis reveals that stable features cover broader lexical/semantic diversity; unstable features focus on narrow, brittle substrings.

Quantitative masking protocols establish that removing unstable features (even at quadruple the stable feature budget to match expected activation mass) causes minimal degradation in explained variance or next-token loss, whereas removing stable features rapidly impairs both.

Figure 3: Masking stable versus unstable features shows stable features account for dominant explained variance and downstream loss.

Geometric and Subspace Analysis

The geometric underpinning is crucial: unstable features, though unreproducible as individual vectors, consistently occupy reproducible low-dimensional subspaces. Effective rank computations reveal unstable feature sets are 20-27% lower-dimensional than stable sets under SAE decoder embeddings. Despite their instability, the subspaces spanned by the unstable features exhibit high cross-seed overlap: the principal subspace from any seed accurately approximates the corresponding subspace in others.

Figure 4: SVD-based explained variance curves demonstrate cross-seed reproducibility of learned subspaces for both stable and unstable feature sets.

Logistic regression classifiers distinguish unstable features in decoder space, but the classification transfers robustly across seeds, confirming the structural nature of instability. This is further probed via a controlled synthetic model comprising a mixture of full-rank ground-truth features and a shared low-rank subspace. SAEs reliably recover the full-rank features with high stability and cosine alignment, but low-rank features align only at the subspace level, reflecting seed-dependent basis selection rather than noise.

Figure 5: Synthetic model confirms full-rank features are stable and well-aligned; low-rank features are unstable but subspace-reproducible.

Feature-Pool Construction and EV-Stability Tradeoff

To operationalize seed-robust SAE construction, the authors propose pooling features from SAEs trained with independent seeds and deduplicating under cosine similarity. Dictionaries built from the most-probable pooled features recover near-maximal explained variance after brief tuning and are dominated by stable features even with modest source pool sizes. Notably, no stability–EV tradeoff is observed in this construction: high stability and high explained variance are attainable simultaneously, contradicting common intuitions about the necessity of such tradeoffs in single-run training.

Figure 6: As pool size grows, unstable-feature fraction decreases and explained variance remains high in most-probable constructions.

Figure 7: Explained variance curves for constructed SAEs illustrate preservation of reconstruction quality with stable feature initialization.

Qualitative and Automatic Interpretation Analysis

Stable features are more interpretable under automatic metrics and LLM-based evaluations. They attract higher detection scores and their explanations more frequently reference structural, compositional, or semantic content. By contrast, unstable features’ explanations center on surface-form or idiosyncratic substrings. Even LLMs can reliably distinguish stable from unstable features using explanation text alone, underscoring the systematic nature of the distinction.

Ablations: Model, Layer, Sparsity, and Random Baseline

The stability dichotomy persists across variations in base model, layer, and dictionary size. TopK and BatchTopK SAE variants are functionally identical in both stability and explained variance; Vanilla SAE is more stable but incurs an EV penalty. More SAE training does not eliminate instability: unstable feature fraction plateaus at a non-zero value even after billions of tokens. Training SAEs on random transformer activations produces high automatic interpretation scores but negligible cross-seed stability, emphasizing the necessity of stability metrics for faithfulness.

Figure 8: Unstable-feature fraction as a function of SAE training tokens shows convergence to a non-zero plateau.

Figure 9: Dead-salmon control: trained versus random transformers yield sharply divergent stability profiles.

Practical and Theoretical Implications

The results demonstrate that SAE instability is systematically structured: unstable features are not mere noise but manifestations of seed-dependent basis ambiguity within reproducible low-rank subspaces. This insight reframes feature interpretability: a single SAE dictionary can obscure relevant cross-seed structure, and traditional auto-interpretation metrics can be misleading in the absence of reproducibility. Pool-based SAE constructions offer practical routes to more stable dictionaries without sacrificing reconstruction fidelity.

From a theoretical standpoint, these findings implicate core neural mechanisms—such as self-attention anisotropy and dimensional collapse (Wang et al., 23 Aug 2025, Godey et al., 2024)—as sources of low-dimensional subspaces conducive to feature non-identifiability. Future directions include explicit identification and disentanglement of these components and the development of objectives or architectural modifications to recover individually stable feature bases within low-rank regions.

Conclusion

This paper rigorously establishes that seed dependence in sparse autoencoder training results from structured basis ambiguity within reproducible subspaces. Stable SAE features are functionally and semantically dominant, while unstable features possess weak marginal impact but encode reproducible, low-rank structure. Pooling features across seeds enables construction of stable SAE dictionaries while retaining high reconstruction quality, challenging the presumed stability–reconstruction tradeoff in mechanistic interpretability. These results clarify the limitations of auto-interpretation and point to refined protocols for interpretable and faithful feature extraction from neural models.

---

References: "Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders" (2606.12138).