PairSAE: Dual Approaches in Sparse Autoencoders

• PairSAE is a term that encompasses two distinct methods in sparse-autoencoder research, one focusing on a pairwise interpretability protocol for language models and the other on a sparse-dictionary approach for protein models.
• The pairwise-matrix protocol variant enhances mechanistic interpretability by examining feature behavior across a two-dimensional intervention space that compares single-feature with joint multi-feature steering.
• The protein structure variant uses an SVD-based token role summary combined with a shared sparse code to jointly reconstruct sequence and pair representations, leading to improved affinity predictions and mechanistic insights.

Searching arXiv for PairSAE and closely related sparse-autoencoder papers. I’ll inspect the PairSAE paper and a few adjacent papers to ground the article in the current arXiv record. PairSAE is a term used in two distinct senses in recent sparse-autoencoder research. In one usage, it denotes a pairwise sparse-autoencoder interpretability protocol for LLMs: instead of labeling a feature from top-activating contexts and validating it with a single steering intervention, the method maps behavior over a two-dimensional matrix whose axes are the steering coefficient and the joint condition of single-feature versus multi-feature steering, with the explicit claim that standard SAE practice inspects only “one corner of the matrix” (Riegler et al., 4 May 2026). In the other usage, it denotes a sparse-dictionary method for pairformer-style protein structure models: pairwise tensors are summarized into token-wise interaction roles, and a shared token-level sparse code is learned that reconstructs both sequence and pair representations (Migliorini et al., 25 Jun 2026). The shared theme is mechanistic interpretability under conditions where conventional SAEs are claimed to be structurally incomplete.

1. Scope of the term and its place in SAE research

The two arXiv usages of PairSAE are related by problem family rather than by a single unified algorithm. One addresses causal-axis identification in SAE feature steering for instruction-tuned LLMs; the other addresses mechanistic interpretability in pairformer architectures for structural biology, where sequence and pair representations coexist and standard SAEs do not transfer cleanly (Riegler et al., 4 May 2026).

Usage of “PairSAE”	Domain	Core claim
Pairwise-matrix protocol	LLM interpretability	Single-feature inspection can mislabel a feature’s causal axis
Pairformer sparse dictionary	Protein co-folding	A shared token-level sparse basis can explain sequence and pair streams jointly

This dual usage matters because “PairSAE” does not refer to a single architectural family across the literature. In the language-model paper, PairSAE is explicitly not a new autoencoder architecture but a better interpretability protocol over existing SAE features. In the structural-biology paper, PairSAE is an architectural method tailored to pairformer-style protein structure models such as Boltz-2. A related protein-SAE line had already scaled SAEs to ESM2-3B, the base model for ESMFold, and adapted Matryoshka SAEs to protein LLMs in order to make structure prediction mechanistically accessible; that work established the motivation for going beyond small sequence-only protein LMs (Parsan et al., 11 Mar 2025).

2. PairSAE as a pairwise-matrix interpretability protocol

The protocol-oriented PairSAE begins from a critique of the field-standard SAE workflow: take a feature, label it from its top-activating contexts, and validate the label by single-feature steering. The paper argues that this procedure can assign a label that is locally accurate yet still misidentify the feature’s causal axis, because the top contexts may expose only one activation regime of a broader direction (Riegler et al., 4 May 2026).

The proposed remedy is a literal pairwise matrix over two axes. The first axis is the steering coefficient $c$, swept over a range rather than tested at a single positive or negative value. The second is the joint condition, comparing intervention on a single feature with intervention on the sum of multiple feature directions. The steering primitive is unchanged:

$$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$

For joint steering, the paper states that it uses the sum of unit decoder directions over a feature set $\mathcal{F}$ with the same scalar $c$. The point is not to alter the SAE itself, but to vary intervention geometry along dimensions that standard practice usually collapses.

The measurements span several classes of probe. Behavioral rates include disclaimer rate, cluster hit rate, and regex degeneration. Coherence is tracked with per-token NLL under the unsteered model. Geometric probes include the residual norm ratio and cosine similarity to baseline. This produces a dose-response and composition-response map rather than a single validation point.

The paper also defines a rank statistic for feature selection from pool differences,

$$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$

with features that distinguish Pool A from both Pool B and Pool C termed Class-1 features. Coefficient ranges are chosen to match residual norms at the steered layer: for Qwen, $\|h\|\approx 1577$ and $c\in\pm 1000$; for Gemma, $\|h\|\approx 772$ and $c\in\pm 400$; for Llama, $\|h\|\approx 35$ and $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$0.

3. Causal findings from the matrix view

The pairwise-matrix protocol is motivated by three headline findings. The first is an inverted U-shape for feature #26221, labeled AI self-disclaimer from its top-activating contexts. Under a coefficient sweep, disclaimer behavior rises and then falls; at $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$1, the model does not simply intensify the usual disclaimer but instead adopts a fluent contemplative-philosopher voice. The paper contrasts this with #22082, which is monotonic, and #2932, which only appears inverted-U because high-magnitude outputs degenerate. The criterion is therefore not mere non-monotonicity, but non-monotonicity with coherence preserved at the turning point (Riegler et al., 4 May 2026).

The second finding concerns three Qwen features, #29108, #26221, #4405, each of which individually steers a philosophy-of-mind style register. Their reported pairwise cosines are $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$2, $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$3, and $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$4. Despite this near-orthogonality, joint suppression of all three at the same coefficient produces a qualitative collapse: recipes, engine explanations, and tyre instructions degrade into placeholder-like, semantically empty skeletons, whereas single-feature suppression at the same magnitude leaves controls relatively intact. The paper interprets this as evidence that the relevant capability is distributed across a small subspace, not localized in any one independently sufficient feature.

The third finding is a matched-geometry comparison between single-feature suppression, joint suppression, and random-direction perturbation. The residual-stream geometry is approximately matched, with reported values around norm ratio $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$5 and cosine $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$6; representative entries include single #29108 at $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$7 with norm ratio $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$8 and cosine $$h_{\text{steered}}^{(t)} = h_{\text{baseline}}^{(t)} + c \cdot \hat w_f, \qquad \hat w_f = \frac{w_f}{\|w_f\|}.$$9, a random direction at $\mathcal{F}$0 with norm ratio $\mathcal{F}$1 and cosine $\mathcal{F}$2, and joint suppression at $\mathcal{F}$3 with norm ratio $\mathcal{F}$4 and cosine $\mathcal{F}$5. Yet the behavioral regimes diverge sharply: single-feature perturbation yields coherent strategy-filler answers, random-direction perturbation yields diverse but coherent substitutions, and joint suppression alone yields placeholder text. The paper’s conclusion is that coherence loss is direction-pattern-dependent, not magnitude-dependent.

The primary demonstrations are on Qwen3-1.7B-Instruct, with replication on Gemma-2-2B-it. The coefficient-axis finding replicates on Gemma with feature #3997. The joint-condition and matched-geometry findings also replicate on Gemma, although the sign and edge of the effect differ because Gemma’s baseline saturation is different, and Gemma exhibits model-specific failure signatures rather than Qwen’s placeholder tokens. Llama-3.1-8B-Instruct appears only as a cross-model case where the Phase 1–4 pipeline locates a top causally responsible feature; the matrix-level tests are not run there.

4. PairSAE as a shared sparse code for pairformer representations

The structural-biology PairSAE starts from a different failure mode. Pairformer-style models maintain both a sequence representation $\mathcal{F}$6 and a pair representation $\mathcal{F}$7 throughout the forward pass. A naïve SAE on pair representations incurs a quadratic blow-up in pairwise features, while a sequence-only SAE can miss concepts distributed jointly across sequence and pair spaces (Migliorini et al., 25 Jun 2026).

The method therefore has three stages. First, it summarizes $\mathcal{F}$8 via an $\mathcal{F}$9-mode SVD / higher-order SVD. The general decomposition is written as

$c$0

and PairSAE specifically uses the mode-1 and mode-2 singular vectors as token roles in pair space. After truncation to the first $c$1 components, token $c$2 receives the interaction-role summary

$c$3

Second, the encoder concatenates the original sequence embedding and this SVD-derived summary:

$c$4

The paper uses a composition of BatchTopK and ReLU as the sparsifier. In the reported experiments, $c$5, $c$6, $c$7, and the latent dictionary size is $c$8, giving a 32× expansion factor relative to the $c$9-dimensional encoder input after concatenation and layer normalization.

Third, the same latent vector $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$0 reconstructs both streams. Sequence reconstruction is standard,

$$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$1

whereas pair reconstruction is factorized into row and column contributions,

$$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$2

This prevents an explicit pairwise latent explosion: a pair feature is reconstructed from the token’s row role and the partner token’s column role, not assigned as an SAE code to each individual $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$3 cell.

Training uses a Matryoshka SAE loss at nested widths $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$4. The full objective is

$$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$5

with $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$6 so that sequence and pair terms are balanced, and $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$7 is used to “revitalize” dead features. To accelerate training, only one $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$8 is sampled per $$s_i = \tfrac{1}{2}\bigl[z(\bar a_A - \bar a_B)_i + z(\bar a_A - \bar a_C)_i\bigr],$$9 in a Monte Carlo approximation. Reported training choices are Adam, learning rate $\|h\|\approx 1577$0, 250,000 steps, minibatches of 2,048 tokens, and 40,000 PLINDER systems. PairSAE is trained and evaluated without MSA input, due to compute constraints.

5. Empirical performance in structural biology and relation to prior protein SAEs

The pairformer PairSAE is evaluated on Boltz-2 activations for PLINDER protein–ligand complexes. The paper describes two PairSAEs at the third recycling step, corresponding to layers 33 and 64, and reports two main evaluation families: linear probing for interpretability and hypothesis generation for binding-affinity prediction (Migliorini et al., 25 Jun 2026).

For probing, individual PairSAE features are tested against UniProt residue annotations, PLINDER system annotations, and PLIP interaction fingerprints using threshold classifiers of the form $\|h\|\approx 1577$1, with both token-level recall / $\|h\|\approx 1577$2 and complex-level recall / $\|h\|\approx 1577$3 reported. The paper gives the fraction of concepts with $\|h\|\approx 1577$4 as follows: ESM2-650M at token 0.8% and complex 19.7%; PairSAE-R3-L32 at token 29.1% and complex 53.2%; and PairSAE-R3-L64 at token 24.0% and complex 49.6%. Qualitative examples include a transmembrane protein feature, a disulfide bond feature, and a protease active-site / dimer partner feature, with figure-caption values of token/complex $\|h\|\approx 1577$5, $\|h\|\approx 1577$6, and $\|h\|\approx 1577$7, respectively.

For affinity analysis, token features are max-pooled to the complex level,

$\|h\|\approx 1577$8

and a sparse linear model is then fit to predict the Boltz-2 affinity value. The reported results are test $\|h\|\approx 1577$9 with 291 nonzero features for R3-L64, and test $c\in\pm 1000$0 with 237 nonzero features for R3-L33. Test Spearman correlations are $c\in\pm 1000$1 for R3-L64 and $c\in\pm 1000$2 for R3-L33. One highlighted R3-L64 feature, feature 2299, activates on complexes with higher predicted affinity. The paper interprets such ligand-activated features as useful for understanding the model’s affinity behavior, while also stating that they were not mapped to specific biochemical concepts because ligand annotations were sparse.

The significance of these results becomes clearer against the immediate protein-SAE background. Related work on ESM2-3B, the backbone used by ESMFold, argued that protein structure prediction becomes mechanistically interpretable only when SAEs are scaled to the model actually driving folding. In that setting, both Matryoshka and TopK SAEs identified 233 concepts with $c\in\pm 1000$3 in Swiss-Prot concept discovery, versus 72–95 for 8M models, and the 3B models outperformed the 8M models on roughly 400 concepts with an average $c\in\pm 1000$4 improvement of about 0.25 (Parsan et al., 11 Mar 2025). The same study also showed that keeping only layer 36 in ESMFold yielded $c\in\pm 1000$5 Å RMSD, while using the SAE reconstruction gave $c\in\pm 1000$6 Å, close to the $c\in\pm 1000$7 Å baseline and far from the $c\in\pm 1000$8 Å full ablation. A steering case study then altered a hydrophobicity-correlated feature and increased predicted SASA of myoglobin by 31.5%, from 8,369.5 to 11,009.3 $c\in\pm 1000$9, at RMSD 2.76 Å, while keeping the input sequence fixed. This suggests why a pairformer-specific method was needed next: once structural interpretability extends beyond sequence-only backbones, the joint handling of sequence and pair streams becomes a first-order requirement.

6. Misconceptions, related methods, and unresolved issues

A common misconception is that PairSAE always names an autoencoder architecture. In the pairwise-matrix paper, it does not: the method is a protocol for interpreting existing SAE features more cautiously, emphasizing coefficient sweeps, joint steering, coherence checks, and matched-geometry controls. Another misconception is that the protein PairSAE is merely an SAE on flattened pair matrices. The paper explicitly presents it as an alternative to such flattening, using an SVD-based token-role summary plus a shared sparse code that reconstructs both sequence and pair representations (Riegler et al., 4 May 2026).

The broader SAE literature supplies nearby but distinct responses to similar problems. Subspace-Aware Sparse Autoencoders (SASA) replace single decoder directions with learned decoder subspaces, enforce block sparsity via Top-$\|h\|\approx 772$0 group gating, and adapt effective rank with a nuclear-norm regularizer. That work is not PairSAE, but it is explicitly described as close in spirit to pair/group/subspace SAE methods because it relaxes the “one latent = one decoder vector” assumption and targets feature splitting in multi-dimensional features (Dalili et al., 4 Jun 2026). Likewise, “Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone” is also not PairSAE: it shifts the intervention locus from decoder columns to encoder weights, constructing an encoder-derived direction $\|h\|\approx 772$1 and projecting embeddings away from it. The relevance is conceptual rather than terminological, since it likewise challenges the standard assumption that a feature’s actionable semantics reside only in a single decoder direction (Bărbălau et al., 13 Sep 2025).

The main unresolved issues are different in the two PairSAE traditions. For the protocol version, the warning is epistemic: a label derived from top contexts may be only a local description of a feature’s broader causal role, and matrix-level tests were not run on the Llama case. For the structural-biology version, the main limitations are explicitly practical and annotational: PairSAE was trained without MSA inputs, the authors note that MSA changes affinity predictions substantially, and ligand-activating features were not mapped to specific biochemical concepts. Taken together, these points indicate that PairSAE is best understood not as a single settled method, but as a cluster of attempts to make SAE-based interpretability more faithful when causal structure is distributed across coefficients, feature combinations, or pairwise model states.