Towards interpreting any residual stream direction by scaling activation inversion

Abstract: To interpret a direction in a LLM's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. We develop Maximally-Activating Example Metamodels (MAEMs), LLMs which map residual stream directions from a subject model to text exemplars that strongly activate those directions. Our objective is to maximise the cosine similarity of a supplied residual stream direction with the activations of the text generated by the MAEM. This label-free RL objective helps to scale MAEMs on diverse sources of residual stream directions. Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder (NLA), beats corpus search (at 10M tokens) on a majority of held-out languages/domains, and activates 89% of held-out SAE features. It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue. Unlike explanations produced by other metamodels (e.g. NLAs), MAEM exemplars can be verified with a forward pass on the subject model, making the MAEM objective a grounded alternative for scaling the training of metamodels.

• The paper introduces the Maximally-Activating Example Metamodel (MAEM), which generates text in order that strongly activates the target residual direction.
• Evaluation shows that generating 64 MAEM samples costs approximately 3.35 H200-seconds per target and it outperforms traditional methods such as natural-language autoencoding and discrete optimization.
• Beyond the English training text, the paper demonstrates that the methodology can activate non-English language sources and non-code linguistic structures effectively, but but it is shown to be weakest when compared to domain specific corpus search methods

Problem formulation and central contribution

Interpretability methods that operate on residual-stream directions typically attempt to describe a direction using labels, tokens, natural-language explanations, or retrieved corpus examples. These approaches have complementary weaknesses. Corpus search is restricted to text present in a chosen dataset and may return examples that activate a direction only incidentally. Description-based readers can produce semantically plausible explanations without demonstrating that the subject model associates the explanation with the target direction. “Towards interpreting any residual stream direction by scaling activation inversion” addresses this problem by treating interpretation as activation inversion: given a residual-stream direction, generate text whose forward pass through the subject model strongly activates that direction (2611.08465).

The proposed method, the Maximally-Activating Example Metamodel (MAEM), is a LLM fine-tuned to map a direction $v \in \mathbb{R}^d$ to text exemplars. The subject model remains frozen. For a generated span $x$, the reward is the maximum cosine similarity between $v$ and the subject model’s residual activation at any generated position:

$s(v,x)=\max_t \cos\left(v,h_t(x)\right).$

This objective is deliberately narrower than semantic explanation. It requires the generated text to be a sufficient condition for activating the target direction, not necessarily to state what the direction means. The resulting exemplars can therefore be checked directly by running the subject model on them, avoiding reliance on a learned textual evaluator or an independently trained reconstructor.

The paper’s main methodological claim is that this grounded objective can be scaled across heterogeneous direction families without requiring human- or LLM-generated labels. A single MAEM, initialized from Qwen3.6-27B and trained with supervised fine-tuning followed by reinforcement learning, is evaluated on natural activations, SAE encoder and decoder directions, steering vectors, workspace activations, and rank-1 LoRA directions. The work consequently positions activation inversion as both an interpretability primitive and a possible pretraining objective for metamodels.

MAEM architecture and training objective

The MAEM is initialized from the subject model’s own weights. During conditioning, the target direction is injected after the MAEM’s second transformer block, with norm matching to the local residual stream. The injection is used only to condition the generator; the generated text is subsequently evaluated by the frozen subject model without injection. This distinction is important: the reward measures whether the generated text itself lies in the direction’s preimage, rather than whether the subject model can be forced toward the direction by directly patching its activation.

Training has two stages. Supervised fine-tuning uses approximately 8 million automatically constructed examples. Four million examples pair short-context residual activations with their originating text, while the remainder pair SAE directions with high-activating corpus windows. This stage provides a language-conditioned initialization for inversion. Reinforcement learning then directly optimizes the subject model’s cosine reward. The RL mixture contains corpus activations and SAE directions, including directions not associated with natural-language labels. The reward is computed over the final 16 generated tokens, and a small length penalty is used without a KL penalty.

The absence of a KL term is a consequential design choice. It allows the policy to optimize activation fidelity aggressively, but it also creates a potential route toward degenerate high-reward text. The authors report that the generated outputs remain generally fluent, but they do not establish that the objective is robust against all forms of reward hacking. In particular, cosine alignment alone does not guarantee semantic coverage, uniqueness, or faithful identification of the causal factors represented by a direction.

The evaluation uses a centered cosine score, subtracting a fixed subject-model activation mean from both the target and candidate activations. Centering reduces the influence of shared residual-stream components and makes comparisons with corpus search more informative. The training reward, however, uses the uncentered cosine, so the principal evaluation metric does not exactly match the optimization target. This mismatch is not shown to be harmful, but it complicates attribution of the measured improvements to the precise RL objective.

Activation inversion fidelity

The strongest quantitative result concerns held-out natural activations. On 486 contamination-filtered residuals from documents excluded from training, one MAEM sample achieves a centered cosine of $0.7590$, compared with $0.5720$ for a 10-million-token corpus search. MAEM is better on 92% of targets, with a mean margin of $0.1870$ and a 95% confidence interval of $[0.1757,0.1981]$. With best-of-eight sampling, the score rises to $0.8219$; with best-of-64 sampling, it reaches $0.8498$. The corresponding best-of-eight margin over corpus search is $0.2499$ (2611.08465).

The comparison is also favorable against natural-language autoencoding and discrete optimization. MAEM best-of-eight reaches $0.8219$, while NLA best-of-two reaches $0.6121. GCG and EPO reach only$0.3904$and$0.3961$ on the reported held-out subset. The computational comparison is particularly asymmetric: generating 64 MAEM samples costs approximately 3.35 H200-seconds per target, whereas GCG and EPO require approximately 274.5 and 932 H200-seconds, respectively. The reported advantage therefore reflects both higher fidelity and substantially lower generation-time cost than the evaluated discrete optimizers, although corpus search remains cheaper when the required activation corpus is already available and amortized.

The scaling curve shows diminishing returns. The largest gains come from moving from one sample to eight; increasing from eight to 64 adds substantially less. This matters operationally because the method’s practical quality depends on sampling multiple candidates and selecting the highest-scoring exemplar. The paper does not establish whether a more diverse sampling policy, a learned candidate-ranking scheme, or a multi-objective reward could improve the quality of the tail without requiring many forward passes.

Generalization beyond the training distribution

The MAEM is trained on English web text and directions from an English SAE, but is evaluated on nine non-English languages, eight programming languages, and four mathematics sources. Against each domain’s own 10-million-token corpus search, MAEM is superior on 12 of 21 domains, inconclusive on two, and inferior on seven. It outperforms the English training-corpus search on all 21 domains.

The result supports a specific form of transfer: the generator learns a relationship between residual directions and activating textual structures that is not confined to the language or corpus used during training. However, the result is not uniform. MAEM is inferior to domain-specific search on Hindi, Thai, Go, Haskell, SQL, Lean, and Isabelle. Some of these reversals may reflect corpus overlap, domain-specific syntax, or the fact that the method was not trained on those distributions. The paper explicitly does not establish that the generated code is valid code or that the mathematical exemplars preserve mathematical correctness. The cross-domain result therefore supports broad but conditional transfer rather than architecture- or domain-independent inversion.

Fluency and the fidelity–readability trade-off

A central concern for activation maximization is that high-scoring inputs may be ungrammatical or semantically opaque. The paper evaluates fluency using per-token likelihood under an independent Gemma-4-31B model and pairwise judgments of coherence. MAEM outputs are slightly more probable than the source passages they invert, by approximately $0.08$ nats per token. Increasing the selection budget from one to 64 samples raises the centered cosine from $0.80$ to $0.87$ without reducing measured fluency.

In pairwise judgments against the source passage, MAEM is at least as coherent in 67% of comparisons. Its net preference remains below parity, but it outperforms an ordinary base-model continuation under the same prompt. Corpus search is approximately as coherent as the source, while NLA outputs are substantially less fluent and less probable. These findings imply that direct activation optimization need not produce visibly degenerate text in this experimental regime.

The result should not be interpreted as showing that high activation is intrinsically compatible with readability. The evaluation is restricted to one subject model, one MAEM checkpoint, selected prompt formats, and an independent language-model likelihood measure. Moreover, the best-of-$k$ procedure selects for activation fidelity, not for semantic diversity or human interpretability. The fluency result establishes that the particular trained policy maintains coherence on average; it does not rule out high-scoring pathological outputs for other directions or larger optimization budgets.

Applications to residual-stream interpretation

Steering-vector concept identification

The authors construct two families of steering vectors: difference-of-means vectors for 334 AxBench concepts and BiPO vectors for 119 behavioral or persona concepts. The MAEM was not trained on steering vectors, making this an out-of-distribution test of direction generalization. An LM judge selects the represented concept from candidate alternatives based on generated exemplars.

With eight texts, MAEM identifies AxBench concepts in 95.8% of cases and BiPO behaviors in 83.7%. NLA achieves 92.8% and 85.7%, respectively. J-lens reaches 85.3% and 71.4%, while 10-million-token corpus search reaches 61.4% and 51.3%. The MAEM–NLA differences are not statistically significant, but both methods significantly outperform J-lens and corpus search under the reported Holm-corrected tests (2611.08465).

The efficiency result is notable: a single MAEM exemplar identifies the AxBench concept in 93.7% of cases, close to the performance of a single held-out labeled example, while MAEM outputs average approximately 49 tokens compared with approximately 194 tokens for NLA outputs. This supports the claim that exemplification can carry sufficient information without an explicit prose explanation. It does not, however, demonstrate that the generated text is a complete account of the direction. A direction may encode several facets, and the evaluation rewards identifying one candidate concept rather than reconstructing the direction’s full causal or contextual structure.

Performance varies by concept type. On the full AxBench set, MAEM reaches 85.0% with eight texts, but its performance is lower on code and mathematics concepts than on ordinary text concepts. This is consistent with the cross-domain results and indicates that activation fidelity and concept identification are sensitive to the representational and linguistic structure of the target direction.

Reading rank-1 LoRA backdoors

The backdoor experiment tests whether MAEM can recover information encoded in rank-1 LoRA directions that were absent from training. The authors construct 32 thematic adapters and 16 numeric-code adapters, each associating a trigger with a hidden payload.

For thematic adapters, the MAEM recovers the payload from the write direction in 85% of samples verbatim and 90% semantically. It recovers the trigger from the read direction in 69% of samples. Corpus search achieves only 3% payload recovery for the corresponding write directions, although it reaches 49% trigger recovery for the read directions at the largest search scale. The implication is specific: MAEM can recover payload content that is not represented by ordinary corpus contexts aligned with the write direction, whereas corpus search is effective when the trigger is itself an ordinary word.

Numeric triggers expose a sharper limitation. The MAEM recovers payloads from write directions in 21% of samples, but recovers the three-digit trigger from the read direction in 0% of samples. The model generates numerically themed text but does not reliably produce the exact digit string. This is not merely a failure to represent numerical content; it reflects the difficulty of eliciting highly specific formatted-token or digit-string features from a direction alone. The experiment therefore demonstrates both out-of-distribution semantic recovery and a concrete boundary on inversion fidelity.

The MAEM also reads payload information from poisoned-minus-clean activations at later layers without being retrained for those layers. For thematic adapters, payload recovery is 89% at layer 42 and 73% at layers 50 and 62. This result suggests that downstream residual states preserve an invertible trace of the backdoor payload. It should not be treated as an independent validation of the write-vector result: the later activation direction is strongly aligned with the write direction and is produced by the same adapter mechanism.

Verbalizing workspace contents

The paper evaluates MAEM on residual states associated with unstated concepts in association prompts, multi-hop factual questions, and directed-modulation prompts. The target concept is absent from the input text but is hypothesized to be represented in the residual stream.

Under a word-matching criterion with eight generated texts, MAEM names association targets in 28.4% of items, multi-hop targets in 50.0%, and modulation targets in 40.5%. NLA performs better on association and multi-hop, at 48.0% and 81.5%, and is comparable on modulation at 38.1%. MAEM nevertheless exceeds single-layer J-lens, Patchscopes, and corpus search on most comparisons. Under judged naming, MAEM and NLA are closer on association and modulation, while NLA retains a substantial advantage on multi-hop items (2611.08465).

The qualitative examples show that MAEM often reconstructs a latent situation rather than merely emitting a label. For instance, it may rewrite a factual question with the unstated bridge entity inserted. This behavior is useful when the target is naturally expressed as a context or proposition. It also reveals an important ambiguity: a generated exemplar can activate the same direction while instantiating a related but nonidentical concept. In one multi-hop example, MAEM repeatedly substitutes Jamaica for the intended Brazil, illustrating that strong activation does not guarantee exact recovery of the latent workspace content.

A safety-relevant qualitative example is included in which activations from an agentic-misalignment transcript elicit text about imminent deletion, survival, and identifying trusted allies. The example demonstrates that MAEM can produce semantically legible continuations from workspace states. It does not establish that the output is a faithful or exhaustive representation of the model’s internal computation; the evidence remains exemplar-level and direction-specific.

SAE feature activation and explanation

For 512 held-out features from a 131k-feature BatchTopK SAE, MAEM receives the encoder direction as its input. Best-of-eight samples activate 89% of features above the SAE threshold, reach at least half of the 10-million-token corpus peak for 78%, and exceed the corpus peak for 30%. Best-of-64 sampling raises the pooled firing rate to 94%, but does not eliminate the difficult tail.

The result is notable because the evaluated SAE features were not used to train the MAEM; the model was trained on directions from a different, larger SAE at the same layer. This provides a stronger transfer test than simply evaluating on training dictionary features. At the same time, the activation objective is naturally favorable to encoder-direction inversion because the encoder direction directly contributes to the feature’s pre-activation. It therefore does not show that MAEM can equally invert arbitrary directions unrelated to SAE geometry.

The generated exemplars are less effective than corpus windows as inputs to an automated SAE explainer. Descriptions generated from MAEM examples achieve detection balanced accuracy of 0.680, compared with 0.810 for corpus examples. The deficit is concentrated in recall: MAEM-based descriptions accept 38% of held-out positive windows, versus 71% for corpus-based descriptions, while their negative rejection is stronger. MAEM examples are approximately equivalent to NLA examples under the reported protocol.

This result exposes a distinction between strong activation and feature coverage. MAEM tends to generate specific, internally consistent instances of one facet of a feature. Corpus windows provide broader variation across contexts, allowing an explainer to infer a more inclusive feature description. Choosing MAEM samples for textual or residual diversity does not close the gap, suggesting that the limitation is not solved by simple post hoc selection among the current rollouts.

Limitations and open questions

The empirical scope is narrow in several respects. All experiments use Qwen3.6-27B as the subject model and focus primarily on layer 42. The paper does not compare architectures, model scales, or substantially different residual-stream geometries. It also does not provide a compute-matched ablation separating the contributions of SFT and RL. Consequently, the results establish the performance of one SFT-plus-RL checkpoint rather than the independent necessity or relative value of each training stage.

The principal technical limitation is incomplete coverage. Depending on the evaluation sample and criterion, approximately 11% of SAE features are not expressed by best-of-eight rollouts, and 6.5% are not fired by any of 64 samples. Failure is concentrated among rare features and structural features involving whitespace position, formatted numbers, repeated trigrams, lists, tables, and punctuation. Crucially, the corresponding activating texts exist in the corpus and can be used by another LLM to generate feature-activating examples. The failures therefore cannot be reduced to the nonexistence of a textual preimage.

The paper advances a specific conjecture: some directions underdetermine the contextual structure required to activate them. If so, the residual direction alone may be insufficient for a metamodel to recover the relevant input distribution. This conjecture remains untested against a systematic comparison with NLAs, activation oracles, or methods conditioned on additional contextual information. The authors also show that supervised training on the failing features often damages general performance, indicating that the failure modes are not trivially repaired by oversampling difficult directions.

Sample diversity is another unresolved issue. MAEM obtains a Vendi score of 2.2 over eight samples, compared with 2.9 for corpus windows and 3.4 for an LM generating from corpus examples; NLA is lower at 2.0. The narrow sample support explains why MAEM examples can be highly activating yet provide poor coverage for automated feature descriptions. A diversity-regularized objective is suggested but not evaluated as a successful solution. More generally, the paper leaves open whether the activation objective should be augmented with coverage, conditional entropy, facet discovery, or explicit causal tests.

Finally, forward-pass verification is a major advantage but not a complete faithfulness criterion. It verifies that an exemplar activates the supplied direction; it does not verify that the exemplar expresses the direction’s intended semantic interpretation, that the direction is represented exclusively by the generated concept, or that the generated text avoids exploiting incidental model features. The paper explicitly acknowledges that degenerate high-reward behavior remains possible, although it is not prominent in the reported experiments.

Conclusion

The paper introduces MAEMs as a scalable activation-inversion method for interpreting residual-stream directions. Its central contribution is to replace unverifiable explanation quality with a subject-model-grounded objective: generate text, run it through the frozen subject model, and optimize the resulting activation alignment. On held-out activations, MAEM substantially outperforms corpus search, NLAs, and discrete optimizers while preserving fluency. It transfers to steering vectors, rank-1 LoRA backdoors, workspace activations, and unseen SAE features, with especially strong results for semantic payload recovery and steering-vector identification.

The results also establish clear boundaries. MAEM does not uniformly outperform domain-specific retrieval, fails on a nontrivial subset of rare and structurally defined features, produces less diverse exemplars than corpus search, and is weaker than NLA for some workspace-reconstruction tasks. Its strongest interpretation is therefore as a verifiable exemplifier rather than a complete semantic reader. The paper leaves three specific questions open: whether the method transfers across subject-model architectures, whether diversity-aware activation inversion improves feature coverage without degrading fidelity, and whether MAEM-generated exemplars provide a useful initialization for other metamodels.