Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

Abstract: Sparse Autoencoders (SAEs) extract interpretable features from LLMs, but standard variants enforce non-negativity, forcing separate latents for diametrically opposed concepts (e.g., "pressure too high" vs. "pressure too low") and wasting dictionary capacity when features are anticorrelated. We propose the Sign-Aware Gated SAE (SA-GSAE): two-sided gated sparsity with signed magnitude and auxiliary supervision. A polarity-sensitive gate selects support on either sign, a signed-magnitude path avoids L1 shrinkage, and an auxiliary reconstruction prevents gate collapse. Bipolar sharing - one latent encoding both signs along a shared direction - is realised via a new Bi-Jump-ReLU activation; parameter accounting shows sign-awareness stays parameter-efficient even when anticorrelated pairs are rare. On real LLM activations across three mid-depth hookpoints on Pythia-1B and SmolLM3-3B (6 cells, 3 seeds), a half-width SA-GSAE at width H strictly Pareto-dominates a full-width Gated SAE at 2H over the entire swept L0 overlap on 3 of 6 cells (both MLP-output hookpoints and resid-mid/Pythia-1B); on the remaining 3 it matches R^2 within 0.025 (max gap -0.008) while cutting dead fraction by 0.35-0.62 absolute. Sweep-geomean dead-fraction reductions are ~100x-500x on MLP-output cells and Pythia-1B resid, ~2x-4x on attention cells and SmolLM3-3B resid. Ablations show the two-sided gate and auxiliary loss are load-bearing (no auxiliary collapses LR to 0.27, 98% dead); tying r_i^+ = r_i^- is indistinguishable (|Delta R^2| = 0.0015), and we recommend this symmetric variant as default. MLP-output gains come from most latents carrying both polarities; on attention, bipolar structure concentrates in a small set of top latents. Full-width SA-GSAE exhibits a reproducible reconstruction collapse at SmolLM3-3B resid that the half-width entirely avoids.

• The paper introduces SA-GSAE, which uses Bi-Jump-ReLU activations, two-sided learnable thresholds, and auxiliary gate supervision to represent both polarities of an anticorrelated feature with one latent.
• The method matches a full-width Gated SAE on the synthetic Polarity Dial benchmark at half the width, achieving reconstruction MSE of 0.009963 versus 0.009956 while avoiding negative-regime collapse.
• Across real LLM activations, half-width SA-GSAE often Pareto-dominates full-width Gated SAEs, reducing dead-latent fractions by up to 15,000×, although gains are weaker at attention hookpoints and under superposition.

Motivation and problem statement

Sparse autoencoders (SAEs) are a standard tool for extracting interpretable feature dictionaries from LLM activations, but the most widely deployed variants (ReLU, Gated, JumpReLU) enforce non-negativity on latent activations. When a semantic axis is naturally bidirectional — "pressure too high" versus "pressure too low" — a non-negative SAE must allocate two latents along antipodal decoder directions, duplicating dictionary capacity without adding geometric expressivity. Classical sparse coding does not impose this constraint, since its coefficients are signed, and recent work such as AbsTopK has begun to challenge non-negativity directly (Zhu et al., 1 Oct 2025). This paper builds on the Gated SAE's separation of support selection from magnitude estimation (Rajamanoharan et al., 2024) and asks whether opposite polarities must occupy separate latents at all.

The SA-GSAE architecture

The Sign-Aware Gated SAE (SA-GSAE) introduces Bi-Jump-ReLU, a signed dead-zone activation. Each latent computes a decoder-aligned projection $t_i(x)$ and a gate pre-activation $\pi_i(x) = \alpha_i t_i(x) + \beta_i$. Two learnable thresholds $\delta_i^+, \delta_i^- \ge 0$ define a dead zone: the gate polarity $s_i(x)$ is $+1$ above $\delta_i^+$, $-1$ below $-\delta_i^-$, and zero otherwise. Outside the dead zone, the activation is a signed ReLU with per-polarity gains $g_i^\pm = \exp(r_i^\pm)$ and a shared magnitude offset; both polarities share one decoder column. Crucially, thresholds decide support only and are not subtracted from active magnitude, preserving the no-shrinkage property of gated objectives.

Training uses a two-sided hinge penalty on the gate plus an auxiliary reconstruction loss through a stop-gradient decoder, which supervises detection separately from magnitude. Ablations show this auxiliary path is load-bearing rather than incidental: removing it collapses Loss Recovered to 0.27, drives dead fraction to 98%, and clamps $L_0$ at roughly 2 regardless of $\pi_i(x) = \alpha_i t_i(x) + \beta_i$0.

Parameter-efficiency accounting

The paper derives a simple width argument: if a fraction $\pi_i(x) = \alpha_i t_i(x) + \beta_i$1 of ground-truth features occur as anticorrelated pairs, sign-awareness reduces required width to $\pi_i(x) = \alpha_i t_i(x) + \beta_i$2, halving the dictionary when $\pi_i(x) = \alpha_i t_i(x) + \beta_i$3. Accounting for four extra scalars per latent (two thresholds, two magnitude scales), net parameter savings require only $\pi_i(x) = \alpha_i t_i(x) + \beta_i$4 — about 0.39% at $\pi_i(x) = \alpha_i t_i(x) + \beta_i$5. The authors are explicit that this is a structural sanity check, not an identifiability theorem, since the true fraction $\pi_i(x) = \alpha_i t_i(x) + \beta_i$6 of paired features is unobservable. They instead measure a calibration-based lower bound $\pi_i(x) = \alpha_i t_i(x) + \beta_i$7 (the fraction of latents demonstrably firing on both signs), which ranges from 0.039 (resid-mid/SmolLM3-3B) to 0.767 (mlp_out-mid/SmolLM3-3B half-width), clearing the threshold by at least $\pi_i(x) = \alpha_i t_i(x) + \beta_i$8 on every cell tested.

Controlled validation (Protocols A–C)

On a synthetic signed-axis benchmark ("Polarity Dial") with deliberately asymmetric sign-conditioned magnitude distributions, SA-GSAE at width $\pi_i(x) = \alpha_i t_i(x) + \beta_i$9 matches the non-negative Gated SAE at $\delta_i^+, \delta_i^- \ge 0$0: reconstruction MSE of 0.009963 versus 0.009956, with near-ideal split-regime calibration slopes ($\delta_i^+, \delta_i^- \ge 0$1, $\delta_i^+, \delta_i^- \ge 0$2). At matched width $\delta_i^+, \delta_i^- \ge 0$3, the standard Gated SAE collapses on the negative regime ($\delta_i^+, \delta_i^- \ge 0$4). A signed soft-threshold baseline shows that signed activations alone are insufficient: its classical $\delta_i^+, \delta_i^- \ge 0$5 shrinkage yields slope errors of ~0.24–0.26 even after correcting for sparsity, supporting the claim that the gated no-shrinkage objective, not signedness per se, drives the gain.

Protocol B establishes honest negative results: under isotropic normals, coordinate anomalies are undetectable by any model under strict compression (recall near the 1% FPR floor, zero coverage), consistent with rotational non-identifiability. Detection succeeds only for feature-direction anomalies aligned to the learned basis, where SA-GSAE achieves recall ≈ 0.999 in both tails with full coverage while non-sign-aware baselines remain near chance. Protocol C confirms geometrically that SA-GSAE consolidates antipodal pairs into single latents, though pair consolidation degrades as superposition increases — a limitation of the mechanism under interference.

Real-LLM benchmark results

The main evaluation trains SAEs at three mid-depth hookpoints (MLP output, attention output, residual stream) on Pythia-1B (layer 8/16) and SmolLM3-3B (layer 18/36), over OpenWebText caches, with 3 seeds per configuration. The headline comparison is a half-width SA-GSAE ($\delta_i^+, \delta_i^- \ge 0$6) against a full-width Gated SAE ($\delta_i^+, \delta_i^- \ge 0$7):

Backbone	Hookpoint	Overlap $\delta_i^+, \delta_i^- \ge 0$8	Strict Pareto dominance	Dead-fraction geomean reduction
Pythia-1B	mlp_out	[15.3, 144.4]	100%	145×
Pythia-1B	attn	[13.7, 97.8]	26%	1.5×
Pythia-1B	resid	[17.9, 129.3]	100%	492×
SmolLM3-3B	mlp_out	[5.4, 127.4]	100%	96×
SmolLM3-3B	attn	[4.8, 116.0]	66%	2.1×
SmolLM3-3B	resid	[21.1, 111.9]	10%	4.0×

On the three MLP-output cells and resid-mid/Pythia-1B, the half-width model strictly Pareto-dominates the full-width Gated SAE across the entire swept $\delta_i^+, \delta_i^- \ge 0$9 overlap, with peak dead-fraction ratios reaching $s_i(x)$0 (resid-mid/Pythia-1B at $s_i(x)$1) and $s_i(x)$2 (mlp_out-mid/SmolLM3-3B). On the remaining three cells, $s_i(x)$3 stays within $s_i(x)$4 while dead fraction drops by 0.35–0.62 absolute. Matched-LR crossings reinforce this: on mlp_out-mid/Pythia-1B, SA-half reaches LR = 0.75 at $s_i(x)$5 with 0.4% dead, versus $s_i(x)$6 with 73.2% dead for Gated-full.

Against a matched-width Gated SAE, SA-half's MLP-output wins persist ($s_i(x)$7), isolating sign-awareness — not width — as the driver there. On attention cells, SA-half trades small $s_i(x)$8 losses (up to −0.008) for large dead-fraction reductions, and is dominated only on resid-mid/SmolLM3-3B. AbsTopK reaches effectively zero dead on MLP and residual cells but fails on attention (50.2% dead at $s_i(x)$9 on attn-mid/Pythia-1B).

The bipolar census explains the hookpoint asymmetry: dictionary-wide bipolar coverage concentrates on MLP outputs and resid-mid/Pythia-1B ($+1$0 up to 0.767), exactly where capacity wins occur, whereas qualitative bipolar structure at attention is confined to a few top-activation latents (12/64 on attn-mid/Pythia-1B, including a latent contrasting education-reform astroturf against Murdoch-owned-media tokens).

Ablations and design recommendations

Tying $+1$1 (symmetric magnitude) is practically indistinguishable from independent per-polarity scaling on real activations ($+1$2), and cell-wise $+1$3 medians agree within 0.05 everywhere; the paper recommends the symmetric variant as default. Dead-zone initialization exhibits a U-shape: initializing thresholds near zero actively harms capacity utilization ($+1$4 gives 27% dead), so $+1$5 is recommended. A hybrid AbsTopK selector with gated magnitude is strictly dominated by full SA-GSAE, indicating that learned two-sided thresholds and auxiliary supervision jointly drive both capacity utilization and reconstruction quality.

A notable pathology: full-width SA-GSAE exhibits a reproducible reconstruction collapse on resid-mid/SmolLM3-3B (MSE up to 6.0, $+1$6 down to −4.3, worst-case LR of 0.175) that the half-width variant entirely avoids ($+1$7 throughout). The authors flag an SA-GSAE-specific threshold-warmup asymmetry as a plausible contributor but have not run a matched-warmup ablation, and offer no mechanistic explanation for why only this cell collapses.

Limitations

The empirical scope covers three mid-depth hookpoints, two backbones, two widths, and three seeds; layer sweeps, instruction-tuned models, models beyond 3B parameters, and widths below $+1$8 (which would stress-test the parameter-efficiency prediction) are out of scope. Residual-stream Loss Recovered saturates arithmetically, so comparisons there rest on $+1$9 and dead fraction. No interpretability or causal-intervention evaluations (e.g., SAEBench-style probing) are performed — the claims are reconstruction-capacity claims only. Baselines exclude Switch and Matryoshka SAEs, and sign-awareness targets sign splitting specifically, not absorption or hedging pathologies. The comparison against AbsTopK retains each method's native encoder parameterization, so the tables should be read as end-to-end recipe comparisons rather than isolated ablations of encoder tying.

Conclusion

SA-GSAE demonstrates that removing non-negativity from gated SAEs, via a two-sided learnable dead zone with signed magnitude and auxiliary supervision, allows one latent to carry both polarities of an anticorrelated axis without reintroducing $\delta_i^+$0 shrinkage. The strongest result is that a half-width SA-GSAE matches or exceeds a full-width Gated SAE on reconstruction while reducing dead fractions by orders of magnitude on MLP-output and Pythia residual cells — with the symmetric-magnitude variant recommended as default. Open questions include the cause of the site-specific full-width collapse, behavior under tighter width budgets, and whether the capacity gains translate into downstream interpretability improvements.