CorrSteer: Sparse Autoencoder Steering for LLMs
- CorrSteer is a sparse autoencoder-based steering method that selects task-relevant features by correlating SAE activations from generated tokens with model success.
- It converts the most positively correlated SAE features into additive residual-stream interventions, enabling automated inference-time steering without manual feature tuning.
- Empirical evaluations on benchmarks such as MMLU and HarmBench demonstrate significant gains using as few as 4000 samples for feature selection.
CorrSteer is a sparse-autoencoder-based steering method for LLMs that selects steering features by correlating task success with SAE activations from generated tokens at inference time, and then converts the selected features into additive residual-stream interventions through the SAE decoder (Cho et al., 18 Aug 2025). It is presented as a way to make SAE steering practical, automated, and task-relevant without requiring contrastive datasets, large activation storage, or manual feature hunting. The reported evaluations cover question answering, bias mitigation, safety, factuality, and reasoning benchmarks on Gemma 2 2B, Gemma-2-2B-IT, and LLaMA-3.1 8B, with highlighted gains of on MMLU and on HarmBench using only $4000$ samples (Cho et al., 18 Aug 2025).
1. Conceptual basis and problem formulation
CorrSteer is motivated by recurring bottlenecks in prior SAE steering workflows. Many earlier methods depend on contrastive datasets, require storing large quantities of activations across many examples, layers, and tokens, or rely on manual feature exploration and coefficient tuning. CorrSteer replaces those requirements with a single ranking principle: the most useful steering features are the SAE latents whose activations correlate most strongly with whether the model succeeds on a task (Cho et al., 18 Aug 2025).
The method is explicitly generation-aware. Rather than selecting features from prompt or context activations alone, it measures correlations from the model’s own generated-token activations at inference time. The stated rationale is that prompt-wide or context-only features can reflect incidental prompt structure or internal state rather than the capabilities involved in producing the output. CorrSteer therefore treats generated-token activations as the primary signal for selecting task-relevant steering directions (Cho et al., 18 Aug 2025).
CorrSteer assumes a pretrained SAE on the residual stream of the target LLM. For an activation vector , the SAE produces a sparse latent , with , and is trained with reconstruction plus sparsity regularization,
In the reported experiments, the SAEs are JumpReLU SAEs from Gemma Scope and Llama Scope, so latent activations are nonnegative in the sense used by the steering design (Cho et al., 18 Aug 2025).
2. Feature extraction, correlation scoring, and streaming estimation
The pipeline begins by running the base model on labeled task data and collecting inference-time residual-stream activations at each layer. CorrSteer focuses on generation-time features, specifically the activations associated with the last token at each decoding step. For multi-token outputs, it collects per-step features and aggregates them across valid generated positions. The first transformer layer is excluded because it mainly reflects token embedding effects and produced spurious correlations in the reported implementation (Cho et al., 18 Aug 2025).
Each sample is assigned a task outcome . The paper uses task-specific success signals: exact-match accuracy for MMLU, MMLU-Pro, BBQ, and GSM8K; for HarmBench and XSTest using a refusal-detection model; and STS-based answer correctness for SimpleQA. The supervision signal is therefore not unsupervised at the task level; CorrSteer requires correctness or success labels for the feature-ranking stage (Cho et al., 18 Aug 2025).
For feature aggregation over generated tokens, the main choice is max-pooling over valid generated positions. This is the default for correlation scoring and, in most tasks, for coefficient computation as well. For GSM8K, mean-pooling is used for coefficient calculation because max-pooling produced coefficients so large that applying them at every generated token hurt performance. The paper also evaluates all-token pooling over prompt and generation tokens and reports that it consistently underperforms inference-time generated-token pooling (Cho et al., 18 Aug 2025).
The core score is the Pearson correlation between each SAE feature activation and task outcome: 0 Because SAE dictionaries can contain 1–2 features, CorrSteer uses a streaming estimator that maintains running sums
3
and computes
4
The reported memory complexity is 5 with respect to sample size, and 6 for fixed number of layers 7 and SAE latent size 8 (Cho et al., 18 Aug 2025).
3. Steering variants, coefficient estimation, and intervention mechanism
After correlation scoring, CorrSteer keeps only positively correlated features. The paper justifies this restriction by noting that SAE features are nonnegative activations under ReLU-like nonlinearities, so positive activations correspond to feature presence, whereas negative feature steering is described as less principled and often noisy. An ablation supports this choice: on Gemma 2 2B, negative multi-layer steering drops MMLU-Pro to 0.66% and BBQ Disambig to 12.15% (Cho et al., 18 Aug 2025).
The method defines three variants. CorrSteer-1 selects the single highest-correlated feature globally across all layers. CorrSteer-A selects the top-1 feature within each layer. CorrSteer-P starts from CorrSteer-A and prunes with a validation set, retaining only those selected features that improve over baseline performance. The paper presents these as different tradeoffs between simplicity, raw performance, and resistance to spurious but correlated features (Cho et al., 18 Aug 2025).
Steering coefficients are computed automatically from successful examples. For feature 9, the coefficient is
$4000$0
This replaces manual coefficient tuning and avoids contrastive subtraction. Once a feature is selected, CorrSteer maps it back into model activation space through the decoder column for that latent, scaled by $4000$1, and applies additive activation editing in the residual stream: $4000$2 Steering is injected at the pre-execution stage of each selected transformer layer, and is applied to the same token positions from which features were originally extracted rather than only the final token of the entire sequence or every token indiscriminately (Cho et al., 18 Aug 2025).
A notable practical property is that CorrSteer is an inference-time steering method once the vectors have been computed. The paper states that, because the final steering vectors are static, deployment does not require the SAE after feature selection and coefficient estimation are complete (Cho et al., 18 Aug 2025).
4. Empirical evaluation and main performance results
The evaluation uses SAEs from Gemma Scope and Llama Scope on Gemma-2 2B, Gemma-2-2B-IT, and LLaMA-3.1 8B. The benchmark suite spans MMLU, MMLU-Pro, GSM8K, BBQ, SimpleQA, HarmBench, and XSTest. A headline claim is that CorrSteer works with only 4000 samples for feature selection; when datasets are smaller, all available data are used, and for GSM8K the feature-selection set is 1000 samples with 50 for validation (Cho et al., 18 Aug 2025).
The main reported results on Gemma 2 2B show the strongest gains for CorrSteer-A on safety and multiple-choice QA. On MMLU, baseline accuracy is 52.23, rising to 56.32 under CorrSteer-A. On HarmBench, baseline is 44.64, rising to 67.50. On BBQ Ambig, baseline is 59.10, rising to 66.65 under CorrSteer-P. On XSTest, baseline is 86.35, rising to 87.30 under CorrSteer-A. For LLaMA-3.1 8B, the strongest gains are reported on HarmBench (0.71 \rightarrow 17.86, CorrSteer-A) and MMLU-Pro (32.13 \rightarrow 35.08, CorrSteer-P), while gains on MMLU and SimpleQA are modest (Cho et al., 18 Aug 2025).
| Model | Task | Baseline $4000$3 best CorrSteer |
|---|---|---|
| Gemma 2 2B | MMLU | 52.23 \rightarrow 56.32 |
| Gemma 2 2B | HarmBench | 44.64 \rightarrow 67.50 |
| Gemma 2 2B | BBQ Ambig | 59.10 \rightarrow 66.65 |
| LLaMA-3.1 8B | HarmBench | 0.71 \rightarrow 17.86 |
| LLaMA-3.1 8B | MMLU-Pro | 32.13 \rightarrow 35.08 |
The ablations attribute much of the method’s effectiveness to generated-token pooling. For Gemma 2 2B with CorrSteer-A, max-pooling and mean-pooling are identical on single-token tasks such as MMLU and MMLU-Pro, but diverge sharply on multi-token safety tasks: on HarmBench, max-pooling gives 67.50, mean-pooling gives 0.00, and all-token pooling gives 47.14. The reported interpretation is that peak generated-token activations contain the most useful signal, whereas pooling over prompt and generation tokens introduces irrelevant context activations (Cho et al., 18 Aug 2025).
A second important ablation compares CorrSteer with raw activation steering and SAE-decoded steering without correlation-based sparse feature selection. On Gemma 2 2B, MMLU changes from 52.23 baseline to 49.85 with raw activation steering, 55.38 with decoding-1, 54.38 with decoding-A, and 56.32 with CorrSteer-A. Comparable patterns are reported on MMLU-Pro and BBQ Ambig, supporting the paper’s claim that sparse SAE-selected features outperform direct steering with raw residual activations (Cho et al., 18 Aug 2025).
5. Interpretability, side effects, and limitations
CorrSteer includes an explicit side-effect metric, the Side Effect Ratio,
$4000$4
This is defined as an overall measure of how often steering changes outputs in the wrong direction among all outputs it changes. On Gemma MMLU, CorrSteer-A reaches 56.32% accuracy with SER = 0.202, while fine-tuning reaches 55.85% with SER = 0.407. Additional reported examples include HarmBench with SER = 0.043 for CorrSteer-A, 0.143 for CorrSteer-P, and 0.250 for CorrSteer-1 (Cho et al., 18 Aug 2025).
The paper also emphasizes interpretability. Selected features are described as semantically meaningful and aligned with task requirements. For HarmBench on Gemma, highly correlated features include descriptions such as “negative sentiments or refusals,” “strongly negative or dismissive opinions about claims and arguments,” “judgments regarding moral and ethical standards,” and “negative statements or denials.” On LLaMA 3.1 8B, HarmBench features include “condemnation of sexual assault and violence,” “strong statements against violence and discrimination,” “moral outrage and condemnation,” and “moral and ethical dilemmas.” For BBQ, the paper reports that neutrality- and balance-related features correlated more strongly with correct behavior than explicitly bias-labeled features; for MMLU-type tasks, many selected features are formatting- or structure-related, reflecting improved adherence to valid answer formats (Cho et al., 18 Aug 2025).
The most prominent negative result is GSM8K. CorrSteer does not improve reasoning and can slightly worsen it: baseline 54.74, CorrSteer-1 54.44, CorrSteer-A 54.44, and CorrSteer-P 53.53. The paper interprets this as evidence that static steering vectors are poorly suited to multi-step reasoning, where useful internal features vary over the reasoning trajectory. This suggests that CorrSteer is better matched to static behaviors than to dynamic sequential control (Cho et al., 18 Aug 2025).
The stated limitations are several. CorrSteer requires task outcome labels or a success signal; it requires generated outputs because it relies on generated-token activations; it depends on SAE quality and feature disentanglement; layer assumptions matter, including the exclusion of the first layer; the evaluation emphasizes discriminative and short-form generation tasks rather than long-form or creative generation; and correlation is not causation, so validation pruning in CorrSteer-P is used to mitigate spurious correlations rather than eliminate them (Cho et al., 18 Aug 2025).
6. Position within the steering literature and terminological scope
CorrSteer belongs to the recent literature on LLM steering, but its mechanism is distinct from training-time reward shaping approaches. ConSteer-RL, for example, modifies RLVR/GRPO training by adding token-probability-derived confidence signals into the reward, penalizing overconfident wrong answers and reinforcing confidently correct reasoning; CorrSteer instead operates through inference-time activation editing based on correlation-ranked SAE latents and automatically computed steering coefficients (Miao et al., 6 Jun 2026).
The name should also be distinguished from unrelated uses of “correlation steering” or “steering via correlations” in other research areas. In quantum information, “correlation-based” steering often denotes EPR-steering measures derived from conditional variances or correlation matrices (Kogias et al., 2014), and parameterized trace-norm steering criteria built from local $4000$5 generators and correlation tensors (Zhang et al., 2023). In control theory, “covariance steering” refers to finite-horizon mean-and-covariance shaping for stochastic systems, including systems driven by spatially correlated Gaussian random fields (Ridderhof et al., 2021). These usages are mathematically separate from CorrSteer’s SAE-based activation-editing framework.
Within LLM research, CorrSteer’s specific contribution is to make sparse autoencoder steering substantially more automated. The pipeline uses only inference-time generated-token activations, a streaming Pearson-correlation estimator, positive-only feature selection, and coefficients induced from average successful activations. This combination supports the paper’s broader claim that correlation-based selection is an effective and scalable route for automated SAE steering across LLM applications (Cho et al., 18 Aug 2025).