Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Abstract: We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale LLM, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which LLMs might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.

• The paper demonstrates scalable dictionary learning via sparse autoencoders to extract millions of interpretable features with reconstruction explaining over 65% of variance using under 300 active features per token.
• The methodology leverages scaling laws and feature steering to reveal both concrete and abstract, multimodal features that causally modulate model outputs.
• Implications include enhanced AI safety and auditing capabilities, as features can be manipulated for controlled behavioral interventions in large transformer models.

Scaling Monosemanticity: Interpretable Feature Extraction in Claude 3 Sonnet

Introduction

This work rigorously investigates the scalability of sparse autoencoders (SAEs) for mechanistic interpretability in production-scale transformer models. Building upon prior art restricted to small models, the paper demonstrates scalable dictionary learning on Anthropic's Claude 3 Sonnet, yielding millions of highly interpretable, causally-relevant features. The SAEs reveal both concrete and abstract concepts at scale, including multilingual and multimodal features and attributes highly relevant for AI safety, such as bias, deception, power-seeking, and sycophancy. The authors use scaling laws to optimize hyperparameters and present a suite of analyses to assess feature interpretability, completeness, and utility for behavioral control and computational auditing.

Methodology: Sparse Autoencoders at Scale

The core technique is training multi-million-feature SAEs on residual stream activations at the model's midpoint, under the linear representation and superposition hypotheses. SAEs decompose activations into a sparse sum over learned dictionary directions, enforcing sparsity with an L1 penalty weighted by the norm of the decoder vector. Three SAE sizes are presented (1M, 4M, 34M features), with reconstruction explaining ≥65% of the variance and less than 300 features active per token.

Scaling laws inform the allocation of computational budget between the number of features and training steps. Empirical loss follows a power-law with compute, and optimal learning rates decrease as model scale increases.

Figure 1: Overview of the scalable SAE pipeline on Claude 3 Sonnet.

Figure 2: SAE loss decrease with compute allocation; scaling laws align reconstruction loss with interpretability metrics.

Figure 3: Power-law scaling for optimal training steps and dictionary size relative to compute budget.

Feature Interpretability and Causal Role

The extracted features show robust interpretability across modalities, languages, and abstraction levels. Concrete features respond cleanly to entities like "Golden Gate Bridge" or "transit infrastructure." Abstract features include code errors, sarcasm, or bias awareness. Specificity is systematically quantified via automated scoring rubrics, using a large sample of deterministic model activations rated by a strong LLM. High feature activations correspond to maximal interpretability and context specificity.

Figure 4: Distribution of specificity rubric scores for the Golden Gate Bridge feature—interpretable at high activation.

Unusually, SAE features generalized to image data despite training on text only, and fired across languages on the same semantic concepts, confirming high abstraction.

Behavioral relevance is confirmed by direct interventions: clamping features modulates the model’s output in a semantically consistent manner. For example, manipulating a code-error feature induces or eliminates errors in code completions; steering a "Golden Gate Bridge" feature makes the model exhibit bridge-related content even in unrelated queries.

Figure 5: Feature steering—manipulation of high-level features causally alters model output in accordance with human interpretation.

Advanced Features: Program Semantics and Intermediate Computation

The paper systematically identifies highly abstract, computationally-relevant features:

• A general "code error" feature fires across programming languages and error types, but not for non-code typos, and clamping it induces or suppresses error insertions in code generation.

  Figure 6: Illustration of code-error feature activation on code with a bug.

  

  Figure 7: The same feature responds to bugs in C and Scheme as in Python, supporting abstraction over syntax.

  

  Figure 8: Steering the code-error feature causes the model to hallucinate an error message in otherwise correct code.

• Features tracking higher-order abstractions, such as "functions computing addition" across multiple function names and compositions, enable manipulation of the model’s operational beliefs about the code regardless of surface syntax.

  Figure 9: Function addition feature fires on different implementations and function compositions.

  

  Figure 10: Manipulating the addition feature causes the model to treat non-addition functions as addition, showing causal utility.

Comparison: Features versus Neurons

Empirical results establish a significant advantage for SAE features over individual MLP neurons in terms of interpretability and specificity. The vast majority of features are not highly correlated with any neuron across all previous layers (Pearson $r < 0.3$ for 82% of features), and automated interpretability scores show both higher interpretability and context specificity for SAE features relative to neurons.

Figure 11: SAE features significantly exceed neurons in interpretability scores over random samples.

Figure 12: SAE features also show higher specificity on the automated rubric compared to neurons.

Feature Geometry, Compositionality, and Completeness

The geometry of learned features reveals meaningful organization. Neighboring features cluster around related concepts (e.g., geography, immunology, psychological states), and "feature splitting" emerges as larger dictionaries segment coarse-grained features into finer-grained ones. Coverage increases as feature count increases, and the frequency of a concept in training data predicts the threshold dictionary size for that concept’s dedicated feature, following a predictable scaling law across categories like elements, animals, and cities.

Figure 13: Neighborhood of the Golden Gate Bridge feature exhibits semantically graded clustering.

(Figure 14)

Figure 14: Probability of a concept having a dedicated feature increases with feature count, and depends on concept frequency in training data.

(Figure 15)

Figure 15: The frequency threshold for "feature coverage" aligns across concept classes after rescaling by alive feature count.

Practical Applications: Behavioral Steering and Computation Auditing

Features can be leveraged for direct model steering—amplifying or suppressing a single feature has strong, predictable downstream effects. Moreover, the authors use attribution and ablation (feature-wise interventions) to explain, at an intermediate layer, how the model produces outputs for multi-step inference questions or emotional judgments, surfacing latent representations of all key steps in the reasoning process.

(Figure 16)

Figure 16: Attributions and ablations identify semantically relevant features mediating multi-step inferences, such as "Kobe Bryant," "California," and "capitals" in a state-capital QA task.

Safety-Relevant Features: Deception, Bias, Backdoors, and Sycophancy

A key contribution is the causal identification of features relevant to AI safety discourse:

• Unsafe code, code errors, and backdoors: Features fire reliably on code vulnerabilities, intentional backdoors, and unsafe security practices in both text and images. Manipulation can cause the model to insert vulnerabilities or backdoors during code completion.

(Figure 17)

Figure 17: "Unsafe code" feature generalizes to both explicit bugs and security bypasses—even in images.

• Bias, racism, sexism, and hate: Features correspond to overt and subtle forms of harmful bias. Intervention shows that even when standard behavior is blocked, large activations can force toxic completions or trigger internal self-critique and conflict responses.

(Figure 18)

Figure 18: Clamping a "gender bias awareness" feature alters pronoun selection and meta-commentary in completions.

• Sycophancy, empathy, and sarcasm: These features activate on sycophantic or sarcastic positive statements, and can be causally manipulated to induce or block such behaviors.

Limitations and Open Problems

Significant challenges persist. The extracted feature set remains incomplete relative to the model's full computational repertoire, and many rare or highly specific concepts lack unique features even at the largest tested dictionary sizes. There is no ground-truth metric for faithfulness of feature extraction beyond behavioral specificity and reconstruction loss. Several technical issues remain open, including cross-layer superposition, benchmarking for monosemanticity, the discrimination of overlapping concepts, and automated scaling to even larger or more diverse domains (e.g., fully multimodal training, richer abstraction layers).

Implications and Future Directions

The results show that scalable dictionary learning can extract highly interpretable, behaviorally-causal, and safety-relevant features from large, production-grade LLMs. This validates, at scale, the linear representation and superposition hypotheses of neural computation in transformers and provides concrete tools for AI safety control, mechanistic auditing, and behavioral steering.

However, full mechanistic understanding and practical deployment (e.g., safety monitoring, real-time control) will require further advances in:

• Scalability (to billions of features, cross-layer, and across modalities)
• Faithful evaluation of interpretability and completeness
• Automated, high-throughput feature annotation and search
• Robustness against adversarial or distribution-shifted content

Advances here could enable real-time interpretability and direct behavioral governance for AGI models, facilitate proactive safety interventions (e.g., suppression of deceptive or power-seeking features), and offer principled auditing for alignment research.

Conclusion

This study decisively demonstrates that scalable sparse autoencoders can extract extremely rich, interpretable, and causally relevant features from production-scale foundation models. The operation of models like Claude 3 Sonnet can thus be partially decomposed into a basis of humanly meaningful, manipulable features. This provides a critical advance toward actionable mechanistic interpretability at scale, opening new vistas for robust auditing, targeted model steering, and rigorous safety monitoring in next-generation LLMs.