---
title: SAE Neuron Scoring Methods
url: https://www.emergentmind.com/topics/sae-neuron-scoring
type: topic
---

# SAE Neuron Scoring Methods

A Sparse Autoencoder (SAE) neuron scoring procedure assigns quantitative measures of “concept alignment,” class relevance, or interpretability at the level of individual SAE latent dimensions (referred to as “neurons”) within a trained model. Such scores are foundational for downstream applications in topic steering, concept erasure, OOD detection, fairness optimization, and interpretation of high-dimensional representations in LLMs, vision models, and recommender systems. SAE neuron scoring methods are diverse, encompassing both unsupervised and task-driven algorithms, metrics based on activation statistics, embedding distances, causal interventions, and downstream effect quantification.

## 1. Scoring by Semantic Alignment: Topic-Relevance in LLMs

The “SAE neuron scoring” framework introduced for topic alignment in large language models assigns each SAE neuron a continuous score, $s_i \in [0,1]$, reflecting its selectivity for semantically aligned prompts relative to a user-specified alignment set $\mathcal{A}_{\text{align}}$ (e.g., domain-specific medical sentences) versus a large reference set $\mathcal{R}_{\text{ref}}$ (e.g., samples from diverse sources) [2506.12576].

The metric is defined by (i) aggregating each neuron's normalized activation mass per prompt, (ii) embedding all prompts with a fixed sentence transformer (e.g., all-MiniLM-L6-v2), and (iii) weighting prompt-to-alignment-set distances by the neuron’s mean normalized activation:

\[
g(i) = \frac{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i \cdot \mathrm{dist}(p,\,\mathcal{A}_\mathrm{align})}{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i}
\]

where $\mathrm{dist}(p,\,\mathcal{A}_\mathrm{align}) = \min_{p' \in \mathcal{A}_\mathrm{align}} \|E(p) - E(p')\|_2$. Min-max normalization across all neurons yields the alignment score:

\[
s_i = 1 - \frac{g(i) - \min_j g(j)}{\max_j g(j) - \min_j g(j)}
\]

Neurons with $s_i \approx 1$ preferentially fire on $\mathcal{A}_{\text{align}}$–like contexts; $s_i \approx 0$ identifies unaligned or polysemantic units.

This scoring enables “Swap” interventions that steer SAE-weighted activations in the LLM forward pass, amplifying topic control for arbitrary alignment sets. Experimental validations demonstrate improved linguistic acceptability and data efficiency compared to fine-tuning, with only modest computational cost [2506.12576].

## 2. Frequency-Weighted Concept Scoring in Text/Image Models

In the context of concept erasure in diffusion models, neuron identification uses a modulated frequency scoring function. Given balanced concept-present and concept-absent prompt pairs, each neuron $i$ receives two scores:

\[
f_i = \sum_{j=1}^N \sum_{t=1}^T \mathbf{1}(Z_{{\rm norm},i,t}^{(j)} > 0), \quad \mu_i = \frac{1}{N T} \sum_{j,t} Z_{{\rm norm},i,t}^{(j)}
\]
\[
s_i = f_i \cdot \mu_i
\]

Only neurons with $s_i^{\text{concept}}>0$ and $s_i^{\text{deconcept}} = 0$ are retained as concept-selective; highest $s_i^{\text{concept}}$ are prioritized. This rigorous frequency-modulated approach allows for surgical, single-neuron erasure while minimizing off-target effects [2509.21008].

## 3. Separability-Based Scoring: Jensen-Shannon Divergence

A distribution-aware method of scoring SAE neurons exploits the concept of activation separability across labeled concepts. For each neuron, the Jensen-Shannon divergence of the neuron’s conditional activation distributions across $k$ classes defines

\[
D_{\text{JS}}(f_{h^l_j \mid c_1},\dots,f_{h^l_j \mid c_k}) = \frac{\sqrt{\mathrm{JSD}(f_{h^l_j \mid c_1},\dots,f_{h^l_j \mid c_k})}}{\sqrt{\log_2 k}}
\]

where $\mathrm{JSD}$ is the entropy-based divergence of the activation densities. Layer- and neuron-level separability $(S^l)$ quantifies how monosemantic a latent is; high $D_{\text{JS}}$ indicates precise class selectivity [2508.15094]. This score directly predicts the ability of partial suppressive interventions—such as Attenuation via Posterior Probabilities (APP)—to erase target concepts with high specificity and minimal collateral performance penalty.

## 4. Task-Grounded and Causal Intervention–Derived Scores

SAE scoring has been extended to adversarial and causal setups that probe the downstream effect of manipulating specific neurons or groups of features on supervised tasks.

- **SHIFT metric:** Measures the normalized improvement in task accuracy when ablating SAE features attributed as spurious (identified by probe attributions and optionally validated by LLM judges). The score,

\[
S_\text{SHIFT} = \frac{A_{\text{abl}} - A_{\text{base}}}{A_{\text{oracle}} - A_{\text{base}}}
\]

assesses the causal responsibility of selected latent features for undesired predictions [2411.18895].

- **TPP metric:** For a multiclass task, evaluates the isolation of concept-specific features by ablating the top-ranked neurons for each class and measuring class-specific probe drops:

\[
S_\text{TPP} = \text{mean}_{i}(A_{i,i} - A_i) - \text{mean}_{i \neq j}(A_{i,j} - A_j)
\]

Higher scores indicate more “disentangled” representations with respect to the supervized targets.

## 5. Outlier and Anomaly Scoring via Structured Transitions

For OOD detection, SAE features are scored via their co-activation and Markov transition profiles across layers [2605.11920]. For each layer, top-$k$ activated neurons are binarized, and co-activation transition probabilities $p_\ell(j|i)$ are computed. Test inputs are assigned per-layer anomaly scores

\[
a_\ell(x) = - \frac{1}{|A_{\ell-1}(x)| \, |A_\ell(x)|} \sum_{i\in A_{\ell-1}(x)} \sum_{j\in A_\ell(x)} \log p_\ell(j|i)
\]

The mean $S(x)$ over layers forms the overall anomaly score. Transition pairs (i.e., neuron-neuron transitions) can be analyzed for semantic interpretation and OOD signature tracking.

## 6. Direct Activation-Based and Embedding-Based Scoring

Other scoring paradigms rank SAE neurons according to class-wise mean activations, frequency of firing, or embedding-similarity to document or prompt representations.

- **Mean activation scoring:** For a class $c$, aggregate the mean latent activation $\bar z_t^{(c)}$ for each neuron $t$ and rank by magnitude [2507.15227]. Localization alignment can then be compared against ground-truth via IoU or mAP.
- **Cohen’s d bias score:** For tasks like fairness intervention, neurons are scored by the standardized difference of their means across synthetic contrastive groups (e.g., “head” vs “tail” users):

\[
d_j = \frac{\mu_{j,\text{Pop}} - \mu_{j,\text{Unpop}}}{\sqrt{(\sigma^2_{j,\text{Pop}} + \sigma^2_{j,\text{Unpop}})/2}}
\]

High $|d_j|$ indicates strong alignment with one attribute.

- **Embedding scoring:** Compare the cosine similarity between neuron explanations and activations in the context of textual prompts for high-throughput triage and clustering of neurons [2410.13928, 2507.08473].

## 7. Practical Considerations and Comparative Performance

SAE neuron scoring schemes vary in regularization, data dependency, and computational demand. Key practices include:

- Use of large and diverse reference sets for stable statistics [2506.12576].
- Explicit min–max normalization to ensure comparability across latent dimensions.
- Exclusion of “dead” neurons based on activation thresholds.
- Application of LLM judges to assess spurious/irrelevant attribution in task-specific pipelines [2411.18895].
- Hyperparameter settings matched to model and downstream task scale; e.g., number of TopK active neurons, reference set size, attributions per class.

Empirical evaluations indicate that distribution-aware separability and task-based ablation scores are most predictive of real-world steerability and concept control [2508.15094, 2411.18895]. Scoring pipelines supporting direct and efficient deployment are prioritized in practical domains, notably steering LLM outputs, safe concept erasure in diffusion, and improving fairness or interpretability in recommender systems [2506.12576, 2601.15122].

---

**References:**

- "Enabling Precise Topic Alignment in Large Language Models Via Sparse Autoencoders" [2506.12576]
- "Domain Restriction via Multi SAE Layer Transitions" [2605.11920]
- "A Single Neuron Works: Precise Concept Erasure in Text-to-Image Diffusion Models" [2509.21008]
- "Evaluating Sparse Autoencoders for Monosemantic Representation" [2508.15094]
- "Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks" [2411.18895]
- "Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders" [2507.15227]
- "From Insight to Intervention: Interpretable Neuron Steering for Controlling Popularity Bias in Recommender Systems" [2601.15122]
- "Automatically Interpreting Millions of Features in Large Language Models" [2410.13928]
- "Evaluating SAE interpretability without explanations" [2507.08473]

Source: https://www.emergentmind.com/topics/sae-neuron-scoring