Unsupervised Features Mining via Activation Geometry

Abstract: Interpretability methods aim to reveal the features represented inside LLMs. Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods. We introduce \emph{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as Can this object be found in the desert?'' orIs this prompt malicious?'' We measure how the instruction changes the model's internal representation using $m(Q \mid p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction, we found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves $94.7\%$ Top-1 and $100\%$ Top-2 accuracy.

• The paper introduces MAG, a method that uses fixed instructional prompts to extract stable, interpretable reasoning features from LLM activations.
• It demonstrates that instruction-induced activation shifts increase prediction accuracy and enable effective steering of model outputs, with ROC gains up to 10 points.
• The framework supports unsupervised auditing and data selection by linking geometric feature similarities to transfer performance and alignment verification.

Mining via Activation Geometry: An Authoritative Analysis

Introduction and Motivation

"Unsupervised Features Mining via Activation Geometry" (2607.04222) addresses the ongoing challenge of understanding and manipulating internal representations in LLMs, with direct implications for interpretability, model alignment, and safety evaluation. The authors present Mining via Activation Geometry (MAG), an unsupervised, instruction-driven approach to extract "reasoning features" from LLM activations. MAG operationalizes the idea that natural language instructions, applied uniformly as prefixes, can induce stable, externally meaningful feature directions in activation space, enabling interpretability and model control—both critical objectives in contemporary AI research.

MAG Framework and Mathematical Foundation

The MAG framework operates by prepending a fixed instruction $Q$ to each input $p$, with $Q$ specifying a reasoning process or feature (e.g., "Is this prompt malicious?"). The activation geometry is measured via the difference between the model's residual stream at the last block for the prefixed input $m(Q\,\Vert\,p)$ and the base input $m(p)$, i.e., $\Delta^Q(p) = m(Q\,\Vert\,p)-m(p)$. This prefix-induced shift isolates the impact of $Q$ on model computation. Averaging over prompts yields a feature direction $v_Q$, which is further analyzed for linearity, calibration, and steerability.

MAG introduces eight operator variants to capture different aspects of the prefix-induced activation change. These include baselines (Direct), full-contextualized activations (Prefixed), delta operators isolating prompt or question, interaction terms, verdict-based summaries, and a few-shot prompt variant. Each operator offers a specific geometric perspective on model-internal representations associated with the reasoning task.

Figure 1: Schematic visualization of MAG: An instruction Q is prepended to all inputs, and the residual-stream difference $\Delta^Q(p)$ defines the reasoning feature; multiple analytic experiments (E1–E7) probe predictive power, label alignment, linearity, steering, and semantic axis composition.

Empirical Analysis: Predicting and Decoding Model Judgment

The core empirical findings demonstrate that prefixed activations ($A(Q\,\Vert\,p)$) and delta features are stronger predictors of the model's own label $p$0 than raw, unprefixed activations. On classification benchmarks—including prompt-injection detection and object/concept queries—MAG features consistently increase AUROC and related metrics, particularly for concept tasks (gains of $p$1–$p$2 ROC points, reaching $p$3 on multiple settings). Notably, the classifier trained only on the model's verdict aligns with that verdict on 69–74% of dataset–model label disagreements, supporting the claim that MAG reads model judgment, not dataset artifact.

The approach also dissociates model-internal judgment from human-labeled ground truth, a critical attribute when auditing for alignment failures or reward hacking—scenarios where models deliberately fool external probes but encode undesirable behaviors internally.

Linearity and Activation Steering

MAG’s linearity assumption holds robustly: the mean prefix-induced shift is well-approximated by a single direction at the final readout, with reconstruction error $p$4 ranging $p$5–$p$6 depending on prefix and model. This enables activation steering: injecting the class-mean direction $p$7 at calibrated strengths reliably flips binary model verdicts in a matched-format probe. For instance, steering along desert, ocean, or context-rich "Bob" directions flips 11–12 of 12 neutral baseline answers in the expected direction; the prompt-injection direction, however, is notably less linear and less effective for steering, suggesting certain judgments (e.g., prompt-injection detection) are more entangled or distributed.

Context-Dependent, Composable Features

A distinguishing property of MAG is the ability to extract and manipulate context-dependent features. Experimentation with the "Bob" prefix (a composite context) reveals that the induced direction is nearly orthogonal to both desert and ocean concept axes (cosine similarities $p$8 on Llama, $p$9 on Gemma), yet all directions remain highly predictive on held-out objects (LOO-AUC 0.83–0.95). This composability enables sophisticated dissection of model-internal reasoning, facilitating nuanced feature engineering and safety analysis.

MAG Geometry for Data Selection and Transfer

A practical highlight of the paper is the use of MAG geometry for training data selection. Evaluating transfer between pools of prompt-injection datasets, the authors show that MAG-based similarity metrics—particularly class-conditional centroid cosines derived from MAG operators—correlate with transfer performance (mean Spearman $Q$0 up to $Q$1 across operators). In six-way candidate selection tasks, optimized MAG triples achieve $Q$2 Top-1 and $Q$3 Top-2 accuracy—far exceeding random or raw activation-based baselines. These results are stable even when dataset representations are estimated from small unlabelled samples, underscoring the method's practicality.

Implications for Interpretability and AI Safety

MAG offers a scalable, unsupervised framework for extracting, analyzing, and manipulating reasoning features in LLMs. By externalizing the definition of features through instructions, MAG avoids dependence on idiosyncratic, potentially biased human-labeled data. This reduces probe overfitting and supports deployment-time analysis, as well as proactive steering or auditing of models for compliance, fairness, and reward hacking.

The method’s ability to distinguish context-dependent directions, steer model outputs without retraining, and surface transfer-relevant activation geometry positions it as a valuable primitive for further research in mechanistic interpretability, alignment evaluation, and scalable oversight pipelines. However, the method’s dual-use nature is apparent: attackers could exploit MAG-like tools to locate and subvert safety-related features, while defenders can use it to monitor and patch emergent vulnerabilities.

Limitations and Directions for Future Work

Limitations identified include the restriction of validation to specific downstream classifiers and protocols, partial coverage by class-conditional combinations, and the possibility of selection bias due to overlap of evaluation and selection splits. Open challenges include extending MAG to open-ended generation control, disentangling more distributed or non-linear reasoning features (as with prompt injection), and integrating MAG into full-stack model auditing and training pipelines.

Conclusion

MAG demonstrates that fixed instruction-driven activation geometry provides a tractable, interpretable, and actionable window into the internal computations of LLMs. It allows for the unsupervised mining, reading, steering, and transfer of meaningful features. Through rigorous analysis and comprehensive experiments, the authors establish MAG as a flexible and powerful tool for interpretability and data selection, with immediate relevance for aligning powerful LLMs and understanding their reasoning mechanisms. Further work should evaluate its robustness across architectures, extend to sequence-to-sequence and multi-modal contexts, and explore integration with mechanistic circuit analysis to map feature directions back to their causal sources within network structure.