SAE Additivity Residuals (SAR)

• SAE Additivity Residuals are metrics quantifying unexplained behavioral variation in neural models, focusing on the deviations in activation contributions after accounting for specified interaction orders in sparse-autoencoder (SAE) features.
• They distinguish between algebraic additivity and behavioral additivity, revealing how the downstream effects of activation contributions might interact through nonlinear computations, not just summing linearly.
• Key applications include evaluating feature interactions in neural models, improving behavioral predictions, and enhancing model interpretability, with practical implications for model debugging and intervention strategies.

SAE Additivity Residuals (SAR) measure the behavioral variation of a neural model that remains unexplained after approximating the joint effects of selected sparse-autoencoder (SAE) features with terms up to a specified interaction order. The framework distinguishes algebraic additivity of an SAE decoder from behavioral additivity: although decoded activation contributions sum linearly, their downstream effects may interact through nonlinear computation. For a behavioral readout and coalition distribution, $SAR_r$ is the normalized squared residual of the optimal order-$r$ approximation, where $r=1$ measures nonadditivity beyond individual feature effects and $r=2$ measures nonadditivity beyond pairwise effects (Ma et al., 15 Oct 2026).

1. Conceptual foundations

An SAE represents an activation using sparse feature coefficients and decoder directions. For an input prompt $x$, let $h_x$ denote the activation at a selected layer and token position, $z_i(x)$ the activation of feature $i$, $d_i$ its decoder direction, and $N_x$ the set of selected active features. The decoded contribution of feature $r$0 is

$r$1

For a coalition $r$2, the model retains the contributions in $r$3 and removes the remaining selected contributions:

$r$4

Equivalently, defining

$r$5

the intervened activation is

$r$6

The remainder $r$7 is fixed across coalitions and includes the SAE reconstruction residual together with all unselected activation content. The full coalition reproduces the original activation, since $r$8.

For a LLM $r$9 and scalar behavioral readout $r=1$0, the resulting coalition response is

$r=1$1

Depending on the experiment, $r=1$2 can be a target-token logit difference, an agreement-logit difference, a pronoun-logit difference, a CounterFact target logit, an indirect-object-identification logit difference, a numerical-comparison score, or classification accuracy. The function $r=1$3 is therefore a behavioral set function, or feature game, assigning a behavioral value to every retained-feature coalition.

SAR concerns this downstream function rather than the decoder identity. With a linear decoder,

$r=1$4

so reconstruction additivity is algebraically immediate. This does not imply that the downstream effects of those contributions are additive. A nonlinear continuation can produce reinforcement, suppression, or context-dependent effects even when the intervened activation itself is a sum of feature vectors.

2. Low-order behavioral approximations

For a selected feature set $r=1$5, the order-$r=1$6 additive function class is

$r=1$7

The coefficient $r=1$8 represents a behavioral term associated with subset $r=1$9. The cases $r=2$0, $r=2$1, and $r=2$2 correspond respectively to a constant model, an intercept with individual feature effects, and an intercept with individual and pairwise effects. The full class is obtained at $r=2$3 for $r=2$4 selected features.

Let $r=2$5 be the distribution over coalitions. The population order-$r=2$6 approximation is the $r=2$7 projection

$r=2$8

where

$r=2$9

The coalition-induced behavioral variance is

$x$0

The order $x$1 specifies the maximum number of features allowed in a behavioral term. Thus $x$2 aggregates pairwise, triple-wise, and higher-order effects, whereas $x$3 aggregates triple-wise and higher-order effects.

If the downstream continuation is affine on all tested activation states,

$x$4

then

$x$5

which belongs to $x$6. Consequently, $x$7. Nonzero first-order SAR therefore reflects nonlinear or context-dependent downstream behavior on the tested intervention states.

3. Definition and theoretical interpretation

Population SAR at interaction order $x$8 is

$x$9

It is the fraction of coalition-induced behavioral variance not explained by the best order-$h_x$0 approximation. The sequence satisfies

$h_x$1

The calibration properties are

$h_x$2

and, for any $h_x$3,

$h_x$4

Thus $h_x$5 is the minimum normalized squared approximation error achievable by an order-$h_x$6 behavioral explanation. The explained-variance fraction is $h_x$7.

The exact interaction-energy interpretation requires independent subset sampling. Each feature is included independently with probability $h_x$8. Define

$h_x$9

The functions $z_i(x)$0 form an orthonormal product basis. If

$z_i(x)$1

then

$z_i(x)$2

Projection onto $z_i(x)$3 removes all terms with $z_i(x)$4, giving

$z_i(x)$5

Therefore, under the product-coalition distribution, SAR is exactly the normalized energy of interactions above order $z_i(x)$6. This interpretation depends on independent feature inclusion; coalition distributions with correlated feature inclusions, such as Shapley-style sampling, do not yield the same orthogonal decomposition.

For a subset $z_i(x)$7, the interaction coefficient can also be written through the finite difference

$z_i(x)$8

where $z_i(x)$9. For a pair,

$i$0

The coefficient is

$i$1

and hence

$i$2

For uniform coalition sampling, $i$3. The pairwise contribution to first-order residual energy is

$i$4

and the pairwise share is

$i$5

when $i$6, with $i$7 otherwise.

4. Relationship to coalition ordering

Squared approximation error does not fully characterize explanation quality. A behavioral explanation may be valuable because it preserves which coalitions produce larger or smaller behavioral effects. For independent coalitions $i$8, define

$i$9

The behavior-weighted ordering error is

$d_i$0

Ordering reversals involving behaviorally distant coalitions receive greater weight through $d_i$1. The ordering bound is

$d_i$2

For the optimal order-$d_i$3 approximation,

$d_i$4

Consequently, small population SAR guarantees small behavior-weighted coalition-ordering error for the optimal order-$d_i$5 explanation. The converse does not necessarily hold: an approximation can preserve many coalition orderings despite retaining nontrivial squared residual energy.

The pairwise share $d_i$6 characterizes the composition of first-order nonadditivity. A high value indicates that most of the residual beyond individual effects is pairwise rather than genuinely third-order or higher. This distinction is important because a positive $d_i$7 does not by itself determine whether adding pairwise terms will substantially improve behavioral prediction.

5. Empirical estimation and evaluation

Exhaustive evaluation of all $d_i$8 coalitions is expensive. The empirical procedure samples coalitions from the product distribution, divides them into fitting and evaluation sets, and estimates an order-$d_i$9 model by least squares:

$N_x$0

If $N_x$1 is the held-out mean squared prediction error and $N_x$2 the held-out behavioral variance, the reported residual estimate is

$N_x$3

with $N_x$4 in the main large-model experiments.

This empirical quantity is a held-out predictive residual rather than exactly the population SAR. Its error contains both structural nonadditivity and finite-sample fitting error:

$N_x$5

The evaluation reports held-out rank fidelity, behavior-weighted ordering error, sign error relative to the full coalition, and selection regret. The main protocol selects $N_x$6 features, samples 384 coalitions per prompt, uses 256 for fitting and 128 for evaluation, and evaluates unseen coalitions. First-order models contain 17 coefficients, while pairwise models contain

$N_x$7

coefficients.

The empirical study reports positive prompt-median $N_x$8 in all 57 model, layer, and behavior combinations examined. Affine controls produce numerical zeros, consistent with the theoretical result that affine downstream behavior is first-order additive.

Pairwise terms improve held-out coalition prediction in every one of the 18 main model–behavior combinations. Average rank fidelity rises from $N_x$9 to $r$000.0272$r$010.0051$r$02\widehat{\mathcal E}_2<\widehat{\mathcal E}_1$$in all 18 settings. The improvement persists across all 20 repeated fitting and evaluation splits.</p> <p>In a combined steering experiment using Llama-3.1 8B, 16 published <a href="https://www.emergentmind.com/topics/corrsteer" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">CorrSteer</a> SAE features, and 1,024 steering coalitions, first-order modeling has held-out residual$$r$03, rank fidelity $r$04, and ordering error $r$05. Pairwise modeling reduces the residual to $r$06, raises rank fidelity to $r$07, and lowers ordering error to $r$08. The pairwise model is not uniformly better in every individual domain, but the aggregate result demonstrates that behavioral nonadditivity also affects combined steering interventions.

6. Coordinate dependence and empirical findings

To test whether behavioral compositionality is a property of the SAE coordinates rather than solely of the selected activation subspace, the study compares learned SAE coordinates with sum-preserving random rotations. For an orthogonal matrix $r$09 satisfying

$r$10

the rotated contribution matrix is

$r$11

Orthogonality preserves the selected subspace, while $r$12 preserves the total selected contribution. The fixed remainder and downstream model are unchanged.

The main comparison uses Gemma-3 4B, Mistral-7B, and Llama-3.1 8B, six behaviors, 100 prompts per model–behavior setting, and eight rotations per prompt, producing 18 model–behavior comparisons. Learned SAE coordinates have lower median $r$13 than rotated coordinates in all 18 settings. Examples include Mistral subject–verb agreement, where the values are $r$14 for SAE coordinates and $r$15 for rotations; Gemma simple copying, with $r$16 versus $r$17; and Llama simple copying, with $r$18 versus $r$19.

Because the subspace, total contribution, fixed remainder, and downstream computation are held constant, the systematic difference indicates that the learned SAE basis supports a lower-order behavioral description than the tested random bases. It does not establish complete feature independence or global additivity. It establishes coordinate-dependent compositionality under the specified interventions.

Reducing intervention magnitude makes the response more additive: halving removed contributions lowers first-order SAR and improves first-order rank fidelity in eight of nine tested settings. Increasing inclusion probability from $r$20 to $r$21 preserves the qualitative pairwise improvement.

Parameter-matched controls replace the 120 pairwise terms in a $r$22 model with 120 randomly selected third-order terms. The pairwise model outperforms the median matched third-order control in held-out rank fidelity and behavior-weighted ordering error in all 18 settings. This indicates that the useful missing structure is specifically concentrated in pairwise interactions rather than being explained by adding an equal number of arbitrary higher-order parameters.

7. Relation to other residual and additivity concepts

SAR is distinct from ordinary SAE reconstruction error. Reconstruction evaluates whether

$r$23

is close to $r$24, whereas SAR evaluates whether downstream behavioral effects of selected decoded contributions are captured by low-order feature terms. An SAE can have low reconstruction error and high behavioral SAR, or an additive decoder and poor reconstruction accuracy.

The notion is also distinct from independent additivity as discussed in MDL-based SAE interpretation. Independent additivity concerns whether an explanation of a joint latent activation can be understood as the sum of separately understandable feature explanations. Its schematic form is

$r$25

The MDL framework treats reconstruction quality as a distortion constraint and description length as a model-selection criterion; it does not define a behavioral additivity residual, pairwise intervention metric, or SAR (Ayonrinde et al., 2024). Lower description length therefore does not necessarily imply lower behavioral SAR.

ReSAEs provide another related but distinct approach. They residualize later-layer activations against affine predictions from earlier selected layers and train later SAEs on the unexplained residual. Their explicit metric, overinteraction (OI), is a cross-entropy-level intervention residual:

$r$26

OI is therefore an instance of an additivity residual for joint multi-layer interventions, but it differs from population $r$27: OI compares a joint intervention with the sum of single-layer intervention effects, whereas $r$28 measures residual energy after projection onto all terms up to interaction order $r$29 (Poduval et al., 27 May 2026).

ReSAEs move OI closer to zero in reported Pythia settings when sparsity is sufficient and improve multi-layer cross-entropy recovery in the tested Pythia and Gemma configurations. The method removes affine cross-layer predictability, but it does not guarantee semantic independence, causal independence, or exact behavioral additivity.

The acronym SAR is also used for unrelated concepts. In “Statistical Agnostic Regression,” SAR denotes a risk-bounding and hypothesis-testing procedure; its residuals are ordinary prediction errors rather than SAE additivity residuals (Gorriz et al., 2024). In multipartite entanglement research, additivity residuals can instead refer to deviations such as

$r$30

which are unrelated to SAE behavior (Zhu et al., 2010).

8. Limitations and scope

SAR is local and intervention-dependent. Its value depends on the selected features, prompt, behavioral readout, representation layer, downstream continuation, coalition distribution, and intervention magnitude. It is not a global scalar property of an SAE dictionary.

The interaction-energy interpretation relies on independent product sampling. The empirical estimator additionally depends on the number and distribution of sampled coalitions, the fitting procedure, and held-out evaluation. The residual estimate combines structural nonadditivity with finite-sample fitting error.

A near-zero SAR indicates approximate additivity for the specified behavioral readout and intervention protocol, not causal independence of the features. Conversely, a positive SAR does not identify a particular mechanism. It may reflect downstream nonlinearities, feature interactions, context dependence, or limitations of the selected low-order approximation.

The MDL framework emphasizes that feature splitting and hierarchical coding can be advantageous when they compress frequent structure, even though they introduce context-specific or dependent representations. Thus compression, sparsity, reconstruction fidelity, semantic interpretability, causal modularity, and behavioral additivity are distinct criteria.

The reported results support a comparatively low-order behavioral description of learned SAE coordinates: individual effects are useful, and pairwise terms often capture much of the remaining structure. They do not support a claim that SAE features have independent downstream effects. Faithful joint explanations and interventions may still require pairwise and higher-order interaction terms.