Dedicated Feature Crosscoders (DFC)

• Dedicated Feature Crosscoders (DFC) are a novel crosscoder design that partitions a sparse latent dictionary into model-specific and shared components, enhancing feature exclusivity.
• The architecture uses decoder masking to enforce exclusivity, ensuring each model is reconstructed only from its dedicated and shared latent features.
• Empirical results reveal that DFCs improve model diffing by isolating behaviorally relevant differences, aiding safety audits and interpretability in neural networks.

Dedicated Feature Crosscoders (DFCs) are a crosscoder architecture for model diffing that hard-partitions a sparse latent dictionary into shared and model-exclusive components, so that interpretable features unique to one model are easier to isolate than in a standard crosscoder. In the main cross-architecture formulation, the latent index set is divided into $I_A$, $I_B$, and $I_S$, corresponding respectively to features dedicated to model $A$, features dedicated to model $B$, and shared features; model $A$ is reconstructed only from $I_A \cup I_S$, and model $B$ only from $I_B \cup I_S$ (Jiralerspong et al., 12 Feb 2026). Later work applied this idea to RL-induced tool use in [Qwen2.5-3B](https://www.emergentmind.com/topics/qwen2-5-3b), arguing that DFC partitioning can concentrate behaviorally relevant differences into a very small, steerable set of exclusive features (Shportko et al., 25 Jun 2026). The acronym is specific to crosscoder model diffing in these works; it is distinct from the federated-learning module called “Domain Feature Corrector,” also abbreviated DFC, in $F^2$DC (Wang et al., 15 Mar 2026).

1. Definition, scope, and conceptual motivation

In the model-diffing literature, DFCs were introduced to address a specific limitation of standard crosscoders: ordinary crosscoders learn a shared sparse latent space for two models, but the optimization is biased toward features that reduce joint reconstruction error across both models. As a result, model-specificity is usually inferred only after training, typically through the Relative Decoder Norm

$I_B$0

which yields only approximate exclusivity (Jiralerspong et al., 12 Feb 2026).

DFCs replace that post hoc heuristic with architectural exclusivity. The defining partition is

$I_B$1

with $I_B$2 reserved for model-$I_B$3-exclusive structure, $I_B$4 for model-$I_B$5-exclusive structure, and $I_B$6 for shared structure. A feature $I_B$7 is strictly exclusive to model $I_B$8 if

$I_B$9

and analogously for model $I_S$0; in DFCs this condition is enforced by construction rather than merely estimated afterward (Jiralerspong et al., 12 Feb 2026).

The underlying motivation is model diffing understood as the analogue of a code diff for neural networks: rather than re-understanding a new model from scratch, the aim is to identify what changed between two models, including “unknown unknowns” not captured by existing evaluations. Cross-architecture diffing makes this especially difficult because the compared models may have different hidden widths, different tokenizers, different coordinate systems, and a mixture of genuinely shared and genuinely exclusive concepts (Jiralerspong et al., 12 Feb 2026). DFCs were proposed precisely for this regime.

A terminological caution is necessary because the acronym is overloaded. In “Domain-Skewed Federated Learning with Feature Decoupling and Calibration,” DFC denotes the “Domain Feature Corrector” inside $I_S$1DC, a local client-side residual CNN for calibrating domain-related features after a Domain Feature Decoupler; that paper explicitly states that it does not use any term like “Dedicated Feature Crosscoders” (Wang et al., 15 Mar 2026).

2. Architectural formulation and exclusivity mechanism

The standard two-model crosscoder setup begins from aligned activations

$I_S$2

with model-specific encoders and decoders. In the DFC variant, the decoder access pattern is masked so that model $I_S$3 is reconstructed only from $I_S$4, and model $I_S$5 only from $I_S$6. The paper gives the objective

$I_S$7

where $I_S$8 denotes latent activations and $I_S$9 is a dead-feature-prevention auxiliary loss (Jiralerspong et al., 12 Feb 2026).

The decisive architectural constraint is that decoder connections to the “wrong” model are zeroed: $A$0 This blocks gradient flow from model $A$1’s reconstruction loss into $A$2-exclusive features, and vice versa. Shared features remain available to both branches. The intended decomposition is therefore explicit: common variance should accumulate in $A$3, while model-unique variance should accumulate in $A$4 or $A$5 (Jiralerspong et al., 12 Feb 2026).

Sparsity in the cross-architecture DFC is enforced with BatchTopK rather than an explicit $A$6 penalty. The reported target is $A$7, with an auxiliary-loss coefficient $A$8. Important tradeoff parameters include the exclusive partition size, described with examples such as 1%, 3%, and 5% of the dictionary for each model, and a sparsity annealing schedule from $A$9 down to $B$0 over 5000 steps in one setup (Jiralerspong et al., 12 Feb 2026).

The RL tool-use study preserves the same three-way conceptual partition but describes the implementation in terms of “A-exclusive, B-exclusive, and shared sub-dictionaries with gradient masking in order to enforce exclusivity.” There the two models are a base Qwen/Qwen2.5-3B and an RL-finetuned chengq9/ToolRL-Qwen2.5-3B, and the DFC is trained on concatenated residual-stream post-MLP activations $B$1 with top-$B$2 sparsity in the encoder (Shportko et al., 25 Jun 2026). This suggests that the central DFC principle is stable across settings even when the exact training loss and application differ.

3. Training protocol, alignment, and implementation regimes

For cross-architecture diffing, DFCs are trained on residual-stream activations from middle layers of paired models. Two main real-model comparisons are reported: Llama-3.1-8B-Instruct versus [Qwen3-8B](https://www.emergentmind.com/topics/qwen3-8b), and [GPT-OSS-20B](https://www.emergentmind.com/topics/gpt-oss-20b) versus Deepseek-R1-0528-Qwen3-8B; for the latter, the paper specifies layer 12 of GPT-OSS-20B and layer 16 of Deepseek-R1-0528-Qwen3-8B, with activation dimensions $B$3 (Jiralerspong et al., 12 Feb 2026).

Because tokenizers differ, the paper aligns activations with a semantic window expansion algorithm: it steps through both token sequences, attempts one-to-one decoded-span matches, greedily expands the shorter side on mismatch, and when decoded text matches, retains the activation from the final token of each matched window. Reported alignment success rates are 99.2% for Llama/Qwen and 99.1% for GPT-OSS/Deepseek, with failures concentrated in special characters, non-standard quotes, emojis, and some multilingual or chat-formatting cases (Jiralerspong et al., 12 Feb 2026).

The training corpus for the cross-architecture DFC uses 100 million token-aligned activation pairs drawn from a 50/50 mixture of FineWeb and LMSYS-Chat-1M. Activations are normalized by scaling their median $B$4 norm to

$B$5

and dynamic masking removes any activation whose norm is more than $B$6 the batch median from the loss for that step. The real-model setup uses dictionary expansion factor 32, total dictionary size $B$7, target sparsity $B$8, Adam with learning rate $B$9, batch size $A$0, 100,000 training steps, 1000-step warmup, bf16 precision, gradient checkpointing, and decoder norm scale initialization at 0.4 (Jiralerspong et al., 12 Feb 2026).

The RL tool-use study uses a same-architecture pair rather than a cross-architecture pair: both models are 36-layer Qwen2.5-3B systems with hidden dimension $A$1. Its DFCs are trained on 40,000 FineWeb samples and 40,000 ToolRL instruction-output pairs. The paper reports a 48-crosscoder sweep comprising 36 DFCs and 12 CrossCoders, varying architecture, dictionary size $A$2, top-$A$3, exclusive share $A$4 for DFCs, and $A$5, with fixed settings of 9,000 steps, batch size 1,024, Adam with $A$6, and $A$7 (Shportko et al., 25 Jun 2026).

An implementation detail of particular interpretive importance is that the RL paper explicitly characterizes the A-exclusive partition not as a perfect sink for RL-specific capability, but as a filter that concentrates the most model-specific residue. That claim is tied to experiments showing that penalizing exclusive features with $A$8 reduces Model A fidelity, rather than leaving performance unchanged (Shportko et al., 25 Jun 2026).

4. Empirical findings and canonical case studies

The original cross-architecture DFC paper presents both toy-model and real-model evidence. In a synthetic setting with 2048 total concepts and 2.5% exclusive concepts per model, DFCs recovered more true exclusive concepts than standard crosscoders and Designated Shared Feature crosscoders, especially in the undercomplete regime, though at the cost of higher false-positive rates. The authors explicitly frame that tradeoff as acceptable for safety auditing, where missing a dangerous feature is worse than surfacing extra candidates (Jiralerspong et al., 12 Feb 2026).

In real-model experiments, the paper reports that DFCs surfaced interpretable model-exclusive features including CCP alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama-3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. The highlighted ideological and safety features all achieved maximum exclusivity score 5 under the paper’s transfer-based metric, and steering evaluations over 30 curated prompts showed strong causal control in the source model with near-zero effect in the comparison model (Jiralerspong et al., 12 Feb 2026).

The cross-architecture results are notable because standard quality metrics did not degrade. The reported comparison between DFC and standard crosscoder is essentially tied on fraction of variance explained, dead feature rate, and detection score:

• FVE: 0.817 vs 0.817
• dead features: 5.0% vs 5.6%
• detection score: 87.78% vs 87.77% The claimed gain is therefore selective rather than global: DFC improves exclusive-feature separation without harming normal crosscoder quality metrics (Jiralerspong et al., 12 Feb 2026).

The RL tool-use study supplies a second canonical case. Across its 48-crosscoder sweep, encode-decode reconstruction improves the RL model’s tool correctness by

$A$9

raising mean tool correctness from 19% to 50.1%. The same joint dictionary also produces passive transfer into the frozen base model, increasing its mean tool correctness by

$I_A \cup I_S$0

a phenomenon the authors call capability spillover. At the same time, format accuracy never spills over: $I_A \cup I_S$1 format $I_A \cup I_S$2 pp for all 48 runs (Shportko et al., 25 Jun 2026).

The paper’s strongest mechanistic result is localization to one feature. In the flagship setting—dfc-D8k-excl10-k45, layer 13—steering one A-exclusive feature, identified as feature $I_A \cup I_S$3, yields

$I_A \cup I_S$4

with 95% CI $I_A \cup I_S$5. This matches the performance of steering all 7 available A-exclusive neurons in that setup. By contrast, the CrossCoder reaches a higher unbudgeted best of $I_A \cup I_S$6 pp only at $I_A \cup I_S$7, whereas the DFC A-exclusive partition saturates at $I_A \cup I_S$8 (Shportko et al., 25 Jun 2026).

The same study reports a cross-layer mean best-cell improvement of

$I_A \cup I_S$9

with 95% CI $B$0, Cohen’s $B$1, paired $B$2 $B$3, and Wilcoxon $B$4, which the paper interprets as evidence that single-neuron saturation is not a layer-13 artefact (Shportko et al., 25 Jun 2026). Its mechanistic interpretation is correspondingly narrow: the dominant A-exclusive features appear to be structural-template detectors for <tool_call>, <response>, parameter boilerplate, and related Tool Interaction patterns, rather than a generic reasoning module.

5. Methodological relatives, critiques, and adjacent variants

DFCs emerged within a broader family of crosscoder methods, and several neighboring papers sharpen both their rationale and their limits. A central precursor is the observation that standard $B$5-trained crosscoders can produce false model-only latents. “Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning” identifies two such failure modes—Complete Shrinkage and Latent Decoupling—and introduces Latent Scaling to diagnose them. The paper argues that a latent with no base decoder direction is not automatically a dedicated fine-tune-specific feature, and reports that the standard L1 crosscoder on Gemma 2 2B suffers heavily from these issues; BatchTopK substantially mitigates them and yields cleaner chat-only latents such as false-information detection, personal-question handling, and multiple refusal-related mechanisms (Minder et al., 3 Apr 2025).

Delta-Crosscoder occupies a closely related design space for narrow fine-tuning. It partitions the latent code into $B$6, uses a dedicated non-shared subspace for fine-tuning-induced variation, masks shared latents out of delta prediction, and adds an explicit delta loss

$B$7

with $B$8. Evaluated across 10 model organisms, it achieves 10/10 causal recovery coverage, compared with 6/10 for DSF and 4/10 for two BatchTopK SAE baselines (Kassem et al., 16 Feb 2026). The paper explicitly reads this as a setting in which ordinary joint reconstruction underweights small, sparse, behaviorally decisive changes. This suggests that DFC-like dedication can be enforced not only by structural partitioning, but also by dedicated objectives and paired contrastive data.

A sharper critique comes from backdoor isolation. “Activation Differences Reveal Backdoors” compares an L1 Crosscoder on concatenated base and fine-tuned activations with a Diff-SAE trained directly on $B$9. At layer 18, Crosscoder achieves BIS $I_B \cup I_S$0 in the LoRA regime and $I_B \cup I_S$1 in the full-rank regime, while Diff-SAE achieves BIS $I_B \cup I_S$2 and $I_B \cup I_S$3 respectively (Kumar, 8 May 2026). The paper interprets the backdoor as a directional activation shift rather than a sparse feature that cleanly emerges in a joint latent basis. A plausible implication is that DFC-style partitioning is not universally sufficient; in some safety settings, the geometry of the signal may favor difference-first representations.

Beyond pairwise model diffing, several papers generalize the crosscoder paradigm in ways that bear on DFC. Cross-snapshot work on pretraining dynamics learns a shared sparse latent space with snapshot-specific decoders, using decoder norms to track when features emerge, persist, or split across checkpoints (Ge et al., 21 Sep 2025). “Crosscoding Through Time” adds Relative Indirect Effects (RelIE) to identify which temporally aligned features are causally important for linguistic behavior (Bayazit et al., 5 Sep 2025). “Sparse Crosscoders for diffing MoEs and Dense models” introduces explicitly designated shared features with tied decoders, though not a full three-way DFC partition; it reports about 87% fractional variance explained and finds that the MoE learns significantly fewer unique features than the dense model (Chaudhari et al., 6 Mar 2026). “Group Crosscoders for Mechanistic Analysis of Symmetry” shows that a crosscoder can be specialized around a transformation group, yielding feature families organized by symmetry rather than by model exclusivity (Gorton, 2024). Together these results place DFCs within a larger movement from generic joint sparse coding toward more structured latent decompositions.

6. Interpretation, limitations, and open questions

DFCs make a representation-level claim, not a capability-level claim. The original cross-architecture paper emphasizes that finding a model-exclusive feature does not mean the other model lacks the concept entirely; rather, one model has a particular sparse direction for that concept, and the other lacks a direct linear analogue in the learned aligned space (Jiralerspong et al., 12 Feb 2026). This point is methodologically important because it limits how far exclusivity can be read as evidence about provenance or genuine novelty.

False positives remain an acknowledged tradeoff. The original DFC paper notes that many exclusive features do not correspond to meaningful model differences and treats DFC as a high-recall prescreening tool rather than a final proof of behavioral difference (Jiralerspong et al., 12 Feb 2026). The RL paper reaches a similar conclusion from another angle: DFC does not perfectly isolate RL-induced tool-use capability, because tool-relevant information still leaks into shared features, causing capability spillover into the frozen base model. That paper therefore characterizes the exclusive partition as a filter rather than a sink (Shportko et al., 25 Jun 2026).

Sensitivity to partition size, seed, and training stochasticity is another recurrent issue. In the cross-architecture paper, broad CCP-alignment-like features were robust, whereas American exceptionalism and finer-grained pro-China subfeatures were less consistently found (Jiralerspong et al., 12 Feb 2026). In the RL paper, the strongest single-feature effect appears in a particular model-layer-configuration combination, even though cross-layer evidence suggests the phenomenon is broader (Shportko et al., 25 Jun 2026). This suggests that DFCs offer a useful inductive bias, but not a uniquely identifiable decomposition.

A final open question concerns when architectural dedication is the right inductive bias at all. The backdoor comparison with Diff-SAE indicates that if the behavior of interest is best modeled as a low-energy displacement vector in activation space, then even an exclusivity-aware crosscoder may be solving the wrong problem unless it builds the differential structure directly into the representation (Kumar, 8 May 2026). Conversely, the success of cross-architecture DFCs, Delta-Crosscoder, and the RL tool-use study indicates that when the target differences are sparse, semantically coherent, and at least partly localizable, explicit dedicated capacity can materially improve feature recovery (Jiralerspong et al., 12 Feb 2026).

Taken together, the literature defines DFCs as a structured crosscoder family whose distinctive contribution is to make exclusivity architectural rather than heuristic. Their main empirical use has been unsupervised model diffing across architectures and post-training regimes, especially where the goal is to surface model-specific ideological, safety, persona, or agentic mechanisms. Their main unresolved problem is not whether dedicated features can exist, but how reliably such features correspond to genuine differences in internal organization rather than artifacts of sparsity, alignment, or representation geometry.