SAE Neuron Scoring Methods

• SAE Neuron Scoring is a method that quantifies the alignment of sparse autoencoder neurons using activation statistics, embedding distances, and causal methodologies.
• It enables precise interventions such as topic steering, concept erasure, and out-of-distribution detection in large language and vision models.
• The approach integrates unsupervised and task-driven techniques to offer actionable insights for controlling model fairness and interpretability.

A Sparse Autoencoder (SAE) neuron scoring procedure assigns quantitative measures of “concept alignment,” class relevance, or interpretability at the level of individual SAE latent dimensions (referred to as “neurons”) within a trained model. Such scores are foundational for downstream applications in topic steering, concept erasure, OOD detection, fairness optimization, and interpretation of high-dimensional representations in LLMs, vision models, and recommender systems. SAE neuron scoring methods are diverse, encompassing both unsupervised and task-driven algorithms, metrics based on activation statistics, embedding distances, causal interventions, and downstream effect quantification.

1. Scoring by Semantic Alignment: Topic-Relevance in LLMs

The “SAE neuron scoring” framework introduced for topic alignment in LLMs assigns each SAE neuron a continuous score, $s_i \in [0,1]$, reflecting its selectivity for semantically aligned prompts relative to a user-specified alignment set $\mathcal{A}_{\text{align}}$ (e.g., domain-specific medical sentences) versus a large reference set $\mathcal{R}_{\text{ref}}$ (e.g., samples from diverse sources) (Joshi et al., 14 Jun 2025).

The metric is defined by (i) aggregating each neuron's normalized activation mass per prompt, (ii) embedding all prompts with a fixed sentence transformer (e.g., all-MiniLM-L6-v2), and (iii) weighting prompt-to-alignment-set distances by the neuron’s mean normalized activation:

$$g(i) = \frac{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i \cdot \mathrm{dist}(p,\,\mathcal{A}_\mathrm{align})}{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i}$$

where $$\mathrm{dist}(p,\,\mathcal{A}_\mathrm{align}) = \min_{p' \in \mathcal{A}_\mathrm{align}} \|E(p) - E(p')\|_2$$. Min-max normalization across all neurons yields the alignment score:

$$s_i = 1 - \frac{g(i) - \min_j g(j)}{\max_j g(j) - \min_j g(j)}$$

Neurons with $s_i \approx 1$ preferentially fire on $\mathcal{A}_{\text{align}}$–like contexts; $s_i \approx 0$ identifies unaligned or polysemantic units.

This scoring enables “Swap” interventions that steer SAE-weighted activations in the LLM forward pass, amplifying topic control for arbitrary alignment sets. Experimental validations demonstrate improved linguistic acceptability and data efficiency compared to fine-tuning, with only modest computational cost (Joshi et al., 14 Jun 2025).

2. Frequency-Weighted Concept Scoring in Text/Image Models

In the context of concept erasure in diffusion models, neuron identification uses a modulated frequency scoring function. Given balanced concept-present and concept-absent prompt pairs, each neuron $i$ receives two scores:

$\mathcal{A}_{\text{align}}$0

$\mathcal{A}_{\text{align}}$1

Only neurons with $\mathcal{A}_{\text{align}}$2 and $\mathcal{A}_{\text{align}}$3 are retained as concept-selective; highest $\mathcal{A}_{\text{align}}$4 are prioritized. This rigorous frequency-modulated approach allows for surgical, single-neuron erasure while minimizing off-target effects (He et al., 25 Sep 2025).

3. Separability-Based Scoring: Jensen-Shannon Divergence

A distribution-aware method of scoring SAE neurons exploits the concept of activation separability across labeled concepts. For each neuron, the Jensen-Shannon divergence of the neuron’s conditional activation distributions across $\mathcal{A}_{\text{align}}$5 classes defines

$\mathcal{A}_{\text{align}}$6

where $\mathcal{A}_{\text{align}}$7 is the entropy-based divergence of the activation densities. Layer- and neuron-level separability $\mathcal{A}_{\text{align}}$8 quantifies how monosemantic a latent is; high $\mathcal{A}_{\text{align}}$9 indicates precise class selectivity (Fereidouni et al., 20 Aug 2025). This score directly predicts the ability of partial suppressive interventions—such as Attenuation via Posterior Probabilities (APP)—to erase target concepts with high specificity and minimal collateral performance penalty.

4. Task-Grounded and Causal Intervention–Derived Scores

SAE scoring has been extended to adversarial and causal setups that probe the downstream effect of manipulating specific neurons or groups of features on supervised tasks.

• SHIFT metric: Measures the normalized improvement in task accuracy when ablating SAE features attributed as spurious (identified by probe attributions and optionally validated by LLM judges). The score,

$\mathcal{R}_{\text{ref}}$0

assesses the causal responsibility of selected latent features for undesired predictions (Karvonen et al., 2024).

• TPP metric: For a multiclass task, evaluates the isolation of concept-specific features by ablating the top-ranked neurons for each class and measuring class-specific probe drops:

$\mathcal{R}_{\text{ref}}$1

Higher scores indicate more “disentangled” representations with respect to the supervized targets.

5. Outlier and Anomaly Scoring via Structured Transitions

For OOD detection, SAE features are scored via their co-activation and Markov transition profiles across layers (Shaheen et al., 12 May 2026). For each layer, top-$\mathcal{R}_{\text{ref}}$2 activated neurons are binarized, and co-activation transition probabilities $\mathcal{R}_{\text{ref}}$3 are computed. Test inputs are assigned per-layer anomaly scores

$\mathcal{R}_{\text{ref}}$4

The mean $\mathcal{R}_{\text{ref}}$5 over layers forms the overall anomaly score. Transition pairs (i.e., neuron-neuron transitions) can be analyzed for semantic interpretation and OOD signature tracking.

6. Direct Activation-Based and Embedding-Based Scoring

Other scoring paradigms rank SAE neurons according to class-wise mean activations, frequency of firing, or embedding-similarity to document or prompt representations.

• Mean activation scoring: For a class $\mathcal{R}_{\text{ref}}$6, aggregate the mean latent activation $\mathcal{R}_{\text{ref}}$7 for each neuron $\mathcal{R}_{\text{ref}}$8 and rank by magnitude (Nakka, 21 Jul 2025). Localization alignment can then be compared against ground-truth via IoU or mAP.
• Cohen’s d bias score: For tasks like fairness intervention, neurons are scored by the standardized difference of their means across synthetic contrastive groups (e.g., “head” vs “tail” users):

$\mathcal{R}_{\text{ref}}$9

High $$g(i) = \frac{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i \cdot \mathrm{dist}(p,\,\mathcal{A}_\mathrm{align})}{\sum_{p \in \mathcal{R}_{\text{ref}}} \mathrm{summ}(p)_i}$$0 indicates strong alignment with one attribute.

• Embedding scoring: Compare the cosine similarity between neuron explanations and activations in the context of textual prompts for high-throughput triage and clustering of neurons (Paulo et al., 2024, Paulo et al., 11 Jul 2025).

7. Practical Considerations and Comparative Performance

SAE neuron scoring schemes vary in regularization, data dependency, and computational demand. Key practices include:

• Use of large and diverse reference sets for stable statistics (Joshi et al., 14 Jun 2025).
• Explicit min–max normalization to ensure comparability across latent dimensions.
• Exclusion of “dead” neurons based on activation thresholds.
• Application of LLM judges to assess spurious/irrelevant attribution in task-specific pipelines (Karvonen et al., 2024).
• Hyperparameter settings matched to model and downstream task scale; e.g., number of TopK active neurons, reference set size, attributions per class.

Empirical evaluations indicate that distribution-aware separability and task-based ablation scores are most predictive of real-world steerability and concept control (Fereidouni et al., 20 Aug 2025, Karvonen et al., 2024). Scoring pipelines supporting direct and efficient deployment are prioritized in practical domains, notably steering LLM outputs, safe concept erasure in diffusion, and improving fairness or interpretability in recommender systems (Joshi et al., 14 Jun 2025, Ahmadov et al., 21 Jan 2026).

---

References:

• "Enabling Precise Topic Alignment in LLMs Via Sparse Autoencoders" (Joshi et al., 14 Jun 2025)
• "Domain Restriction via Multi SAE Layer Transitions" (Shaheen et al., 12 May 2026)
• "A Single Neuron Works: Precise Concept Erasure in Text-to-Image Diffusion Models" (He et al., 25 Sep 2025)
• "Evaluating Sparse Autoencoders for Monosemantic Representation" (Fereidouni et al., 20 Aug 2025)
• "Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks" (Karvonen et al., 2024)
• "Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders" (Nakka, 21 Jul 2025)
• "From Insight to Intervention: Interpretable Neuron Steering for Controlling Popularity Bias in Recommender Systems" (Ahmadov et al., 21 Jan 2026)
• "Automatically Interpreting Millions of Features in LLMs" (Paulo et al., 2024)
• "Evaluating SAE interpretability without explanations" (Paulo et al., 11 Jul 2025)