Geometry-Adaptive Explainer (GAE)

• The paper introduces a training-free approach that realigns decoder subspaces via sparse autoencoders to restore causal faithfulness under distribution shift.
• It minimizes the faithfulness gap by aligning the top‑r decoder subspace with the OOD-active subspace, using orthogonal Procrustes and closed-form ridge refit.
• Empirical evaluations on models like GPT-2 and Pythia-1.4B demonstrate improved nAOPC and nComp metrics with minimal computational overhead.

Searching arXiv for the cited papers to ground the article in current records. Geometry-Adaptive Explainer (GAE) is a training-free method for dictionary-based mechanistic interpretability under distribution shift. It is introduced as a post-hoc adaptation procedure for sparse autoencoders and transcoders trained on in-distribution activations, with the aim of restoring causal faithfulness when out-of-distribution (OOD) shift rotates the subspace that the model actively uses (Lim et al., 21 May 2026). The method formalizes this failure mode as a geometric misalignment between the explainer’s decoder subspace and the OOD-active subspace, defines the resulting discrepancy as the faithfulness gap, and reduces it by realigning the decoder to unlabeled OOD activations while preserving the original feature structure (Lim et al., 21 May 2026). In a broader sense, the phrase “geometry-adaptive explainer” has also been used conceptually for explanation methods that adapt attributions to curved predictive-distribution manifolds in evolving graphs or to local gradient geometry in image models, but the capitalized acronym GAE in the strict sense refers to the 2026 dictionary-adaptation method rather than to a graph auto-encoder (Lim et al., 21 May 2026, Liu et al., 2024, Rahman et al., 2022).

1. Definition and problem formulation

GAE is formulated for a fixed, pretrained network with hidden activation at a chosen layer $h(X)\in\mathbb{R}^d$. The underlying explainer is dictionary-based: sparse autoencoders and transcoders learn a decoder or dictionary $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$ and encoder $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$ that reconstruct $h(X)$ or the MLP output at the layer. The generic form is

$$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$

where $\sigma$ is a sparsifying nonlinearity such as ReLU or Top-K (Lim et al., 21 May 2026).

The central claim of the method is that an explainer trained on in-distribution (ID) activations reflects the ID geometry of hidden states, whereas OOD shift changes the second-moment structure of activations and thereby rotates the model’s active subspace (Lim et al., 21 May 2026). For environment $e\in\{\mathrm{ID},\mathrm{OOD}\}$, the second moment is

$M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$

For a target rank $r$, the active subspace projector is $\Pi_e=U_eU_e^\top$, where $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$0 holds the top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$1 eigenvectors of $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$2. The explainer subspace is $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$3, where $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$4 are the top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$5 left singular vectors of $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$6 (Lim et al., 21 May 2026).

The paper defines the faithfulness gap as the Frobenius distance between projectors,

$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$7

and equivalently,

$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$8

where $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$9 are principal angles between the subspaces (Lim et al., 21 May 2026). This makes subspace misalignment the operative mechanism by which OOD faithfulness degrades.

The method distinguishes between hidden-space and causal criteria. The theoretical surrogate is the OOD projection loss

$W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$0

where lower is better (Lim et al., 21 May 2026). The experimental evaluation then connects improved geometry to causal logit-level metrics rather than treating subspace alignment as an end in itself.

2. Geometric mechanism of OOD failure

The geometric picture is explicit: distribution shift changes $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$1 to $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$2, rotating $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$3 away from $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$4. Since well-trained ID explainers empirically satisfy $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$5, the decoder remains tied to the old geometry and becomes misaligned under OOD (Lim et al., 21 May 2026). GAE treats this as a structural rather than purely optimization-related failure.

This framing yields a precise interpretation of faithfulness degradation. The OOD projection loss for any rank-$W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$6 projector $W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$7 decomposes as

$W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$8

where the first term is irreducible and the second is explainer-dependent (Lim et al., 21 May 2026). With OOD eigengap

$W_{\mathrm{enc}}\in\mathbb{R}^{k\times d}$9

the explainer-dependent term is equivalent, up to spectral constants, to the square of the faithfulness gap: $h(X)$0 Accordingly, minimizing $h(X)$1 directly controls the reducible component of OOD faithfulness loss (Lim et al., 21 May 2026).

The same paper also bounds the misalignment itself by second-moment shift. If the ID eigengap at rank $h(X)$2 is

$h(X)$3

then

$h(X)$4

This establishes a Davis–Kahan-type relation between statistical shift and geometric degradation (Lim et al., 21 May 2026). A plausible implication is that GAE is most naturally suited to regimes in which OOD shift is well captured by changes in second moments and low-rank subspace rotation.

3. Algorithmic construction

GAE takes as input the ID dictionary $h(X)$5, the fixed encoder $h(X)$6, unlabeled OOD activations $h(X)$7, regularization hyperparameters $h(X)$8, and rank $h(X)$9. It outputs an adapted dictionary $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$0 aligned to the OOD-active subspace; the encoder is kept fixed to preserve feature selection semantics (Lim et al., 21 May 2026).

The empirical OOD second moment is

$$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$1

The OOD-active subspace is given by the top-$$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$2 eigenvectors $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$3 of $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$4, with projector $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$5 (Lim et al., 21 May 2026).

The optimization problem is

$$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$6

with $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$7 and $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$8 frozen (Lim et al., 21 May 2026). The first term targets reconstruction on OOD activations, the second targets the faithfulness gap, and the third prevents catastrophic deformation of learned features.

The algorithm has two steps.

First, GAE performs subspace rotation via orthogonal Procrustes. Let $$\hat{h}(X)\;=\;W_{\mathrm{dec}}\,\sigma\!\bigl(W_{\mathrm{enc}}\,h(X)+b_{\mathrm{enc}}\bigr)+b_{\mathrm{dec}},$$9 be the top-$\sigma$0 left singular vectors of $\sigma$1. It parameterizes a rotated decoder by

$\sigma$2

This ensures $\sigma$3 and hence $\sigma$4 for any orthogonal $\sigma$5, so the faithfulness gap becomes $\sigma$6 up to estimation (Lim et al., 21 May 2026). The chosen rotation solves

$\sigma$7

With

$\sigma$8

the closed-form solution is

$\sigma$9

This is the geometry-alignment step in the strictest sense (Lim et al., 21 May 2026).

Second, GAE performs a constrained decoder refit while keeping the encoder fixed. Feature activations are

$e\in\{\mathrm{ID},\mathrm{OOD}\}$0

Using centered statistics

$e\in\{\mathrm{ID},\mathrm{OOD}\}$1

define

$e\in\{\mathrm{ID},\mathrm{OOD}\}$2

The closed-form solution is

$e\in\{\mathrm{ID},\mathrm{OOD}\}$3

$e\in\{\mathrm{ID},\mathrm{OOD}\}$4

Inside the OOD-active subspace, the ridge level is $e\in\{\mathrm{ID},\mathrm{OOD}\}$5; outside, it is $e\in\{\mathrm{ID},\mathrm{OOD}\}$6 (Lim et al., 21 May 2026). The paper also allows an optional stability interpolation,

$e\in\{\mathrm{ID},\mathrm{OOD}\}$7

with $e\in\{\mathrm{ID},\mathrm{OOD}\}$8 and $e\in\{\mathrm{ID},\mathrm{OOD}\}$9 by default unless otherwise noted (Lim et al., 21 May 2026).

4. Theoretical properties

The main theorem states that GAE improves over the unadapted ID explainer on OOD data. Assuming $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$0, $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$1, and $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$2, then

$M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$3

This gives a guaranteed improvement in the hidden-space surrogate loss (Lim et al., 21 May 2026).

A related corollary upper-bounds the OOD degradation of the ID explainer by the second-moment shift: $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$4 The proof sketches are described as using projector calculus, Ky Fan’s principle, and Davis–Kahan’s $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$5 theorem (Lim et al., 21 May 2026).

The causal evaluation is operationalized separately. The experiments use normalized comprehensiveness at feature budget $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$6,

$M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$7

normalized AOPC averaged over budgets $M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$8,

$M_e\;:=\;\mathbb{E}_{X\sim P_e}[h(X)h(X)^\top].$9

and

$r$0

where higher is better for nComp and nAOPC, and closer to $r$1 is better for $r$2 (Lim et al., 21 May 2026). The paper’s theoretical stance is that hidden-space geometry acts as a surrogate for these causal metrics, while the experiments test whether the surrogate transfers.

5. Empirical evaluation

The empirical study covers GPT-2 Small ($r$3, layer 8) and Pythia-1.4B ($r$4, layer 15), with transcoders and Top-K SAEs, all trained on ID activations (Lim et al., 21 May 2026). The OOD settings are temporal shift on FineWeb, domain shift on Edgar, and adversarial shift on HaluEval; all are reported to exhibit measurable second-moment shifts and corresponding subspace rotations (Lim et al., 21 May 2026).

GAE is training-free and uses approximately $r$5 OOD activations with no gradients. The reported wall clock is approximately $r$6s for GPT-2 and approximately $r$7s for Pythia-1.4B (Lim et al., 21 May 2026). This is contrasted with training-based baselines: Finetune uses $r$8M tokens, while Retrain, SAEBoost, and FaithfulSAE use $r$9M tokens (Lim et al., 21 May 2026).

For the transcoder on GPT-2 Small, the reported results are as follows.

OOD setting	nAOPC	nComp	$\Pi_e=U_eU_e^\top$0
FineWeb	0.960	1.494	0.0167
Edgar	0.981	1.618	0.0009
HaluEval	0.871	0.963	0.0014

These values are reported as best across methods for GPT-2 Small transcoders, with the Edgar result also compared against Retrain at $\Pi_e=U_eU_e^\top$1, and the HaluEval result compared against SAEBoost at nComp $\Pi_e=U_eU_e^\top$2 and (|\Delta\mathrm{CE}|=0.0212$$(<a href="/papers/2605.21849" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Lim et al., 21 May 2026</a>).</p> <p>For Pythia-1.4B transcoders, GAE attains best nAOPC and nComp across all three shifts;$$\Pi_e=U_eU_e^\top$3 is best on Edgar ($\Pi_e=U_eU_e^\top$4) and HaluEval ($\Pi_e=U_eU_e^\top$5), while on FineWeb it is $\Pi_e=U_eU_e^\top$6, slightly above Finetune’s $\Pi_e=U_eU_e^\top$7 (Lim et al., 21 May 2026). For GPT-2 Small SAEs, GAE leads on all nine cells across the three shifts and three metrics; the paper gives HaluEval as an example with nComp $\Pi_e=U_eU_e^\top$8 and (|\Delta\mathrm{CE}|=0.0017$\Pi_e=U_eU_e^\top$91.155$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$000.0218$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$01=1.946$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$02 (Lim et al., 21 May 2026).

The geometric diagnostics are particularly central to the paper’s interpretation. Principal angles show that GAE’s top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$03 decoder subspace aligns to the OOD-active subspace at approximately $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$04 degrees, whereas Fixed and Finetune retain large $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$05–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$06 gaps (Lim et al., 21 May 2026). A step ablation further shows that Step 1 closes the gap to $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$07 but can lower nComp, for example from $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$08 to $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$09, while Step 2 recovers causal coherence, for example to nComp $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$10, with a small remaining gap of approximately $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$11 (Lim et al., 21 May 2026). A case study in direct logit attribution reports that, with the same encoder and same top-3 features, rotating the decoder increases class specificity, illustrated by nationality tokens changing from Fixed $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$12 to GAE $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$13 (Lim et al., 21 May 2026).

6. Practical use, assumptions, and limitations

The practical recipe is intentionally lightweight. The paper states that $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$14k–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$15k unlabeled OOD activations typically suffice, with approximately $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$16 used in the experiments (Lim et al., 21 May 2026). No labels and no gradients are required. The main costs are forming $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$17, performing an $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$18-dimensional eigendecomposition, solving an $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$19 Procrustes problem, and optionally doing the closed-form ridge refit (Lim et al., 21 May 2026).

The computational structure is correspondingly simple. Subspace estimation costs $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$20 to form $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$21 naïvely, though it can be done in streaming with a running second moment and randomized eigen-solvers for the top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$22 space. Forming $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$23 costs $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$24, and the SVD of the $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$25 matrix is cheap. Step 2 computes $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$26 and $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$27; for very large $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$28, the paper suggests Step 1 only, diagonal or block-diagonal approximations to $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$29, or restricting Step 2 to rows in $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$30 with iterative solvers (Lim et al., 21 May 2026).

The hyperparameters are also described concretely. The rank $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$31 should capture the dominant OOD shift directions; the paper states that even $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$32–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$33 can work well, with nComp stable at least $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$34 across $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$35–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$36 on GPT-2/HaluEval (Lim et al., 21 May 2026). Defaults for $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$37 and $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$38 are around $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$39–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$40 for GPT-2, with larger $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$41 for Pythia when needed, for example $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$42–$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$43 (Lim et al., 21 May 2026). The interpolation parameter is $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$44 by default; setting $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$45 skips Step 2 if OOD sample size is tiny or dictionary size is very large (Lim et al., 21 May 2026).

Several assumptions and limitations are explicit. GAE assumes the main OOD effect is captured by a rotation or warping of second-moment structure, namely a linear subspace shift (Lim et al., 21 May 2026). Strongly non-linear or feature-wise sparse perturbations beyond second moments are not directly modeled. Reliable estimation of the top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$46 OOD eigenspace requires enough OOD samples. Rank truncation ignores information below $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$47, so choosing $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$48 too small can underfit and choosing it too large can add noise, although the paper reports robustness across $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$49 (Lim et al., 21 May 2026). Step 2 can be expensive for very large $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$50, which is why the paper often skips it for large SAEs and relies on Step 1 (Lim et al., 21 May 2026).

The preservation of semantics is treated as a design constraint rather than a secondary convenience. The encoder stays fixed, keeping sparse feature activations and signs unchanged. Step 1 is an orthogonal rotation in the top-$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$51 subspace, preserving angles and scales within that subspace. Step 2 regularizes toward the rotated dictionary and against leaving $W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$52, which the paper presents as a way to keep semantics coherent (Lim et al., 21 May 2026).

7. Terminological scope and related geometry-adaptive explainers

The acronym GAE has an important ambiguity. In graph representation learning, “GAE” often denotes “Graph Auto-Encoder,” but the method under discussion does not train auto-encoders for graph embedding; its “geometry-adaptive” designation refers to adaptation to the geometry of hidden-state subspaces under distribution shift (Lim et al., 21 May 2026). This distinction is necessary because the same initials are common in another part of the graph-learning literature.

The 2026 GAE paper also sits within a broader family of explanation methods that adapt to geometry, although those methods use the term only conceptually. In "A Differential Geometric View and Explainability of GNN on Evolving Graphs" (Liu et al., 2024), the main algorithm is referred to as AxiomPath-Convex rather than GAE, but it is described as geometry-adaptive because it embeds predictive distributions on a manifold with a curved Fisher metric, models evolution as smooth curves on that manifold, and selects a unique sparse curve through a strictly convex program (Liu et al., 2024). In that setting, the geometry resides in information geometry of predictive distributions over evolving graphs rather than in second-moment alignment of internal dictionaries.

Similarly, "Geometrically Guided Integrated Gradients" (Rahman et al., 2022) is not named GAE, but it can be viewed as a geometry-adaptive explainer because it explores gradient-ascent trajectories launched from multiple points along the integrated-gradients path and aggregates the strongest gradient responses (Rahman et al., 2022). Its geometry is local input-space gradient flow, not subspace rotation. The method is defined by

$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$53

with attribution

$W_{\mathrm{dec}}\in\mathbb{R}^{d\times k}$54

taken element-wise (Rahman et al., 2022).

These related uses make the phrase “geometry-adaptive explainer” broader than the specific capitalized acronym. A plausible implication is that the unifying theme across these works is not a common architecture but a common explanatory principle: explanations should conform to the intrinsic geometry relevant to the target phenomenon, whether that geometry is induced by Fisher information on predictive distributions, by local gradient flow in input space, or by second-moment subspace structure in hidden representations (Liu et al., 2024, Rahman et al., 2022, Lim et al., 21 May 2026). In the narrow and formal sense, however, Geometry-Adaptive Explainer denotes the post-hoc, training-free dictionary realignment method for faithful mechanistic interpretability under OOD shift introduced in 2026 (Lim et al., 21 May 2026).