Exemplar Partitioning for Mechanistic Interpretability

Abstract: We introduce Exemplar Partitioning (EP), an unsupervised method for constructing interpretable feature dictionaries from LLM activations with $\sim 10^3\times$ fewer tokens than comparable sparse autoencoders (SAEs). An EP dictionary is a Voronoi partition of activation space, built by leader-clustering streamed activations within a distance threshold. Each region is anchored by an observed exemplar that serves as both its membership criterion and intervention direction; dictionary size is not prespecified, but determined by the activation geometry at that threshold. Because exemplars are observed rather than learned, dictionaries built from the same data stream are directly comparable across layers, models, and training checkpoints. We characterise EP as an interpretability object via targeted demonstrations of properties newly accessible through this construction, plus one head-to-head benchmark. In Gemma-2-2B, EP dictionary regions are interpretable and support causal interventions: refusal in instruction-tuned Gemma concentrates in a region whose exemplar ablation can collapse held-out refusal. Cross-checkpoint matching between base and instruction-tuned dictionaries separates the directions preserved through finetuning from those introduced by it. EP regions and Gemma Scope SAE features decompose activation space differently but agree on a shared core: $\sim$20% of EP regions match an SAE feature at $F_1 > 0.5$, and EP one-hot probes retain $\sim$97% of raw-activation probe accuracy at $\ell_0 = 1$. Nearest-exemplar distance provides a free out-of-distribution signal at inference. On AxBench latent concept detection at Gemma-2-2B-it L20, EP at $p_1$ reaches mean AUROC 0.881, +0.126 over the canonical GemmaScope SAE leaderboard entry and within 0.030 of SAE-A's 0.911, at $\sim 10^3\times$ less build compute.

• The paper introduces Exemplar Partitioning, an unsupervised method that builds activation space dictionaries via leader clustering and Voronoi partitions.
• It demonstrates that EP retains nearly 97% of raw activation probe accuracy while reducing computational cost by roughly 10^3 times compared to sparse autoencoders.
• The method enables causal manipulation and robust out-of-distribution detection by anchoring dictionary regions with immutable, real observed activations.

Exemplar Partitioning: A Unsupervised Approach for Efficient Mechanistic Interpretability

Methodological Foundations

The paper introduces Exemplar Partitioning (EP), an unsupervised method for extracting interpretable dictionaries from LLM activation spaces via leader clustering. Unlike sparse autoencoders (SAEs), which require extensive training, fixed dictionary sizes, and bundle reconstruction with sparsity objectives, EP constructs its dictionary as a Voronoi partition by streaming activations and anchoring each region with a real observed exemplar. The dictionary size emerges in accordance with the geometry of activation space rather than being imposed a priori.

The clustering utilizes centered cosine distance on the unit sphere, with calibration based on the percentile of pairwise distances within a corpus. This percentile, denoted $p$, governs the resolution: lower $p$ yields finer partitions with more regions, while higher $p$ produces coarser dictionaries. Each activation is normalized relative to the corpus center, and new regions are spawned only when activations are not within the threshold distance of existing exemplars. Importantly, exemplars are immutable once selected, enabling direct comparisons across layers, checkpoints, and even models.

Empirical Characterization and Comparative Analysis

EP dictionaries are shown to be locally interpretable and causally actionable. Regions in the Gemma-2-2b model are content- and function-coherent, and both content (e.g., ordinal regions) and function (e.g., code position) axes recur across partitions. Empirical investigations demonstrate substantial agreement between EP and SAE methods: approximately 20% of EP regions (at $p = 10$) attain $F_1 > 0.5$ in token overlap with SAE features. This confirms a strong shared core, though EP and SAE diverge outside this intersection—EP captures broad density-based regions, SAE splinters them into linearly separable, sparse directions.

EP achieves nearly all linearly-decodable identity measured by probing, with the $p = 10$ one-hot code retaining $\sim97\%$ of raw activation probe accuracy. Notably, EP does this at $\sim10^3\times$ less computational cost than the SAE dictionaries, dropping the need for gradient steps and vast training budgets.

Behavioural Localisation and Causal Manipulation

EP’s efficacy extends to explicit behavioral localization. Refusal behavior in instruction-tuned Gemma can be concentrated in a uniquely loaded region, and ablation of the region's exemplar reduces held-out refusal rate by as much as $-0.96$ at $p = 12$, while matched non-refusal regions yield null effect. Exemplar directions are more causally precise for ablation than mean-member directions, highlighting the utility of real observed activations as anchors.

Additionally, the nearest-exemplar distance offers a nonparametric out-of-distribution (OOD) signal: activations from random tokens or underrepresented distributions (e.g., Bulgarian Wikipedia) are systematically further from EP dictionary exemplars, with the gap increasing as partition resolution tightens and decreasing with layer depth.

Dynamics Across Layers, Domains, and Training Checkpoints

EP saturation sizes quantify the representational complexity inherent to activation distributions within domains and layers. Chat inputs saturate EP dictionaries at larger sizes with greater growth across network depth; code inputs are consistently more compact. Saturated region counts correlate with representation complexity at each layer and domain.

Cross-checkpoint matching elucidates training-induced drift in activation space. Hungarian matching of EP dictionaries across base and instruction-tuned Gemma models highlights substantial re-anchoring, with only a small subset of persistent universal regions (e.g., mathematical tasks) remaining. Instruction tuning fragments the previously unitary final-position representation, promoting content-discriminating axes and anchoring refusal along decision-time regions.

Stability, Information Retention, and Limitations

Leader clustering via streaming introduces seed dependence, but region stability is quantifiable using the log-concentrated effective sample size ($p$0). Regions with high member count and coherence are reproducible across seed shuffles. EP's lossy compression discards nearly all raw coordinates but preserves nearly all linearly-probeable label identity under extremely tight one-hot encoding constraints.

EP is currently committed to centered cosine geometry; the role of magnitude, non-cosine metrics, and hierarchical activation structure is left to further work. Streaming order remains a source of instability for large or uniformly represented regions, and exploratory experiments with alternative exemplar selection protocols are proposed.

Practical and Theoretical Implications

EP sets a new standard for unsupervised feature discovery and mechanistic interpretability in LLMs, achieving compelling interpretability, causal manipulability, cross-checkpoint commensurability, and domain-resolved saturation, all with dramatically reduced computational requirements. The practical utility of EP for concept identification, OOD detection, and model behavior steering is well-demonstrated.

Theoretically, EP exposes the density-vs-linear-separability duality in representation learning, providing a vantage point to interrogate the geometry of activation space without the constraints of reconstruction objectives or fixed dictionary sizes. EP serves not only as a feature dictionary but as a direct measurement of activation space geometry and its evolution under training.

The implications extend to efficient model-wide inventorying, precise causal intervention, and rigorous assessments of representational drift—a substantial advance in transparency and interpretability of large networked systems.

Conclusion

Exemplar Partitioning is a rigorous, tractable, and interpretable object for unsupervised analysis of LLM activation spaces. It preserves most linearly-decodable information, enables efficient region-local causal manipulation, and produces dictionaries commensurable across architectures and training checkpoints. EP’s geometric construction, density-aware commitment, and efficiency open several promising avenues for interpretability research, future metric and protocol extensions, and real-time large-scale activation analysis. Its dual contribution as an interpretability tool and a geometric assay of activation structure is poised to facilitate both practical and theoretical progress in AI transparency and mechanistic understanding.