Dedicated Feature Crosscoders in LLM Analysis

• Dedicated Feature Crosscoders (DFCs) are architectural extensions that partition the latent feature space into exclusive and shared subsets, enabling unsupervised isolation of model-specific behaviors.
• They employ sparsity regularization and a revival auxiliary loss to maintain interpretable, compact codes, achieving high reconstruction metrics and LLM-judge accuracy.
• Empirical evaluations demonstrate that DFCs enable precise causal discovery and targeted steering of LLM capabilities, thereby improving model diffing and behavioral analysis.

Dedicated Feature Crosscoders (DFCs) are an architectural extension of sparse dictionary crosscoders designed for unsupervised isolation of features unique to individual neural network models, especially LLMs. Originating in the context of model diffing—the process of comparing internal representations across models—DFCs explicitly partition the crosscoder feature space to enhance identification and interpretability of genuinely model-exclusive behaviors. This approach allows for rigorously causal discovery, localization, and mechanistic steering of differences in capabilities, alignment, and behaviors between LLM checkpoints, finetunes, or architectures (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026, Bayazit et al., 5 Sep 2025).

1. Formal Structure and Definition

A Dedicated Feature Crosscoder extends the standard crosscoder/autoencoder by introducing a hard partition of the latent feature indices into three disjoint sets: features exclusive to model A, features exclusive to model B, and features shared by both.

Let $A$ and $B$ be two models, with activation dimensions $d_A$ and $d_B$. The DFC learns encoders $W_e^A\in\mathbb{R}^{d_A\times m}$, $W_e^B\in\mathbb{R}^{d_B\times m}$ and decoders $W_d^A\in\mathbb{R}^{m\times d_A}$, $W_d^B\in\mathbb{R}^{m\times d_B}$. The set of feature indices $I = \{1,\ldots, m\}$ is partitioned as $I=I_A\cup I_B\cup I_S$, where $B$0 are A-exclusive, $B$1 are B-exclusive, and $B$2 are shared features.

DFC structurally enforces exclusivity: for $B$3, $B$4; for $B$5, $B$6. During training, encoder and decoder gradients for model-exclusive features only flow through their respective models. The effect is that a feature can only participate in reconstructing the activations of one model, never both (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026).

The DFC loss is:

$B$7

where $B$8 is an auxiliary revival loss addressing dead features, and $B$9 (Jiralerspong et al., 12 Feb 2026).

2. Architectural Innovations and Optimization

DFC augments the standard sparse autoencoding crosscoder by:

• Partitioning the feature space before training: typically, 3–10% of features are allocated exclusively to each model, and the majority remain shared (e.g., $d_A$0 / $d_A$1 / $d_A$2 split).
• Zeroing decoder columns and masking encoder gradients according to feature partitions.
• Employing sparsity regularization using BatchTopK activation masking and $d_A$3 penalties to encourage compact, interpretable codes.
• Incorporating a revival auxiliary loss targeting dead (never activated) features by reconstructing high-error cases with inactive feature encoders, thus maintaining feature diversity during learning (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026).

The loss function can include partition-specific sparsity penalties, such as penalizing exclusive feature usage to determine whether exclusives act as pure "sinks" or as "filters" that concentrate genuinely model-specific signals (Shportko et al., 25 Jun 2026).

3. Interpretability and Mechanistic Discovery

DFC-induced exclusive features can be causally interpreted through max-activation analysis and feature-level steering experiments. For example, adding or subtracting a decoder vector corresponding to an exclusive feature to a model's activation induces a behavioral change in that model only, offering strong evidence that the feature is both mechanistically sufficient for the underlying behavior and unique to that model.

This property is central to unsupervised model diffing, as it allows for both post hoc identification of behavioral differences (e.g., alignment, safety, tool use) and causal validation via targeted activations at inference time (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026).

4. Empirical Evaluation and Key Results

Empirical studies across published works demonstrate several core findings:

• DFCs maintain parity with standard crosscoders and designated shared-feature variants on reconstruction metrics such as fraction of variance explained ($d_A$4), dead feature rate ($d_A$5), and interpretability detection scores ($d_A$6 LLM-judge accuracy) (Jiralerspong et al., 12 Feb 2026).
• In both synthetic and real-world LLM comparisons, DFCs recover a richer and higher-recall set of model-exclusive features relative to baselines, concentrating exclusivity scores more tightly at the "maximal exclusivity" end of behavioral evaluation.
• Quantitative steering experiments reveal large, model-specific behavioral shifts upon activation of exclusive features: e.g., steering a CCP-aligned feature in Qwen raises its alignment score from $d_A$7 to $d_A$8 with no effect in the paired Llama model; steering a copyright refusal mechanism in GPT-OSS 20B directly modulates refusal rates, selectively affecting only GPT-OSS (Jiralerspong et al., 12 Feb 2026).
• In the RL finetuning domain, DFCs localize tool-calling capability in Qwen2.5-3B to a minimal subset of exclusive features: steering a single A-exclusive neuron yields a $d_A$9 percentage point gain in tool-correctness (95% CI $d_B$0), matching the effect of steering the entire exclusive set (Shportko et al., 25 Jun 2026).
• "Capability spillover" is observed: shared features can passively transfer some RL-induced capabilities (e.g., $d_B$1 pp tool-correctness) to the base model when activations are routed through a DFC, suggesting shared representations can re-enable dormant behaviors in frozen models (Shportko et al., 25 Jun 2026).

5. Application to Training, Pretraining, and Behavioral Tracking

DFC methodology is extensible to three-way and longitudinal cross-model mappings, where it reveals the dynamics of feature emergence, consolidation, and obsolescence across LLM pretraining. In checkpoint-diffing studies, DFCs paired with causal attribution metrics—especially the Relative Indirect Effect (RelIE)—accurately localize when and where specific linguistic abstractions manifest, are refined, or disappear (Bayazit et al., 5 Sep 2025).

Examples include phase transitions in syntactic abstraction (e.g., irregular plural agreement) and the consolidation of language-specific to crosslingual features in multilingual models. The method demonstrates architecture-agnostic applicability (Pythia, OLMo, BLOOM) and scales linearly with crosscoder parameter count, though memory and compute costs grow with both LLM and crosscoder sizes.

6. Comparative Analysis and Limitations

Comparative baselines include standard crosscoders (with or without post hoc exclusivity heuristics) and designated shared-feature models. DFCs exceed these in isolating meaningful, causally interpretable, and mechanistically sufficient exclusive features, as evidenced by clustering analyses, higher exclusivity concentration, and improved recall in toy models with ground-truth concept partitions (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026).

Limitations include sensitivity to partition choices (e.g., too early checkpoints may lack interpretable exclusives), the need for careful alignment of activation spaces, and potential computational scaling challenges for very large models or dense checkpointing. Integrated gradients used for causal effect estimation are approximations, and manual feature annotation introduces some human subjectivity.

A notable implication is the risk of capability spillover: deployment of DFCs may inadvertently re-enable capabilities (e.g., tool use) in base models at runtime, raising safety and deployment considerations (Shportko et al., 25 Jun 2026).

7. Impact and Significance in Model Diffing and Interpretability

Dedicated Feature Crosscoders establish a robust framework for unsupervised, architecture-agnostic model diffing, enabling precise surfacing of unknown unknowns in large model behaviors. The tight integration of architectural partitioning, sparsity regularization, causal attribution (via activation steering and RelIE), and empirical validation sets DFCs apart from prior art and permits fine-grained, scalable, and interpretable comparison of models across versions, architectures, and training regimes. The approach is applicable to critical domains including safety alignment, detection of emergent behaviors, tracking pretraining dynamics, and runtime control of agentic capabilities (Jiralerspong et al., 12 Feb 2026, Shportko et al., 25 Jun 2026, Bayazit et al., 5 Sep 2025).