---
title: Dedicated Feature Crosscoders (DFC)
url: https://www.emergentmind.com/topics/dedicated-feature-crosscoders-dfc
type: topic
---

# Dedicated Feature Crosscoders (DFC)

Dedicated Feature Crosscoders (DFCs) are a crosscoder architecture for model diffing that hard-partitions a sparse latent dictionary into shared and model-exclusive components, so that interpretable features unique to one model are easier to isolate than in a standard crosscoder. In the main cross-architecture formulation, the latent index set is divided into \(I_A\), \(I_B\), and \(I_S\), corresponding respectively to features dedicated to model \(A\), features dedicated to model \(B\), and shared features; model \(A\) is reconstructed only from \(I_A \cup I_S\), and model \(B\) only from \(I_B \cup I_S\) [2602.11729]. Later work applied this idea to RL-induced tool use in `Qwen2.5-3B`, arguing that DFC partitioning can concentrate behaviorally relevant differences into a very small, steerable set of exclusive features [2606.26474]. The acronym is specific to crosscoder model diffing in these works; it is distinct from the federated-learning module called “Domain Feature Corrector,” also abbreviated DFC, in \(F^2\)DC [2603.14238].

## 1. Definition, scope, and conceptual motivation

In the model-diffing literature, DFCs were introduced to address a specific limitation of standard crosscoders: ordinary crosscoders learn a shared sparse latent space for two models, but the optimization is biased toward features that reduce joint reconstruction error across both models. As a result, model-specificity is usually inferred only after training, typically through the Relative Decoder Norm
\[
\mathcal{R}_i^A = \frac{\|\mathbf{d}_i^A\|_2}{\|\mathbf{d}_i^A\|_2 + \|\mathbf{d}_i^B\|_2},
\]
which yields only approximate exclusivity [2602.11729].

DFCs replace that post hoc heuristic with architectural exclusivity. The defining partition is
\[
I = I_A \sqcup I_B \sqcup I_S,
\]
with \(I_A\) reserved for model-\(A\)-exclusive structure, \(I_B\) for model-\(B\)-exclusive structure, and \(I_S\) for shared structure. A feature \(i\) is strictly exclusive to model \(A\) if
\[
\|\mathbf{d}_i^B\|_2 = 0,
\]
and analogously for model \(B\); in DFCs this condition is enforced by construction rather than merely estimated afterward [2602.11729].

The underlying motivation is model diffing understood as the analogue of a code diff for neural networks: rather than re-understanding a new model from scratch, the aim is to identify what changed between two models, including “unknown unknowns” not captured by existing evaluations. Cross-architecture diffing makes this especially difficult because the compared models may have different hidden widths, different tokenizers, different coordinate systems, and a mixture of genuinely shared and genuinely exclusive concepts [2602.11729]. DFCs were proposed precisely for this regime.

A terminological caution is necessary because the acronym is overloaded. In “Domain-Skewed Federated Learning with Feature Decoupling and Calibration,” DFC denotes the “Domain Feature Corrector” inside \(F^2\)DC, a local client-side residual CNN for calibrating domain-related features after a Domain Feature Decoupler; that paper explicitly states that it does not use any term like “Dedicated Feature Crosscoders” [2603.14238].

## 2. Architectural formulation and exclusivity mechanism

The standard two-model crosscoder setup begins from aligned activations
\[
\mathbf{X}^A \in \mathbb{R}^{N \times d_A}, \qquad \mathbf{X}^B \in \mathbb{R}^{N \times d_B},
\]
with model-specific encoders and decoders. In the DFC variant, the decoder access pattern is masked so that model \(A\) is reconstructed only from \(I_A \cup I_S\), and model \(B\) only from \(I_B \cup I_S\). The paper gives the objective
\[
\mathcal{L}_{DFC} = \sum \left( \|\mathbf{X}^A - \mathbf{F}_{I_A \cup I_S}\mathbf{W}_{d,A}^{excl+shared}\|_2^2 + \|\mathbf{X}^B - \mathbf{F}_{I_B \cup I_S}\mathbf{W}_{d,B}^{excl+shared}\|_2^2 \right) + \alpha \mathcal{L}_{aux},
\]
where \(\mathbf{F}\) denotes latent activations and \(\mathcal{L}_{aux}\) is a dead-feature-prevention auxiliary loss [2602.11729].

The decisive architectural constraint is that decoder connections to the “wrong” model are zeroed:
\[
\mathbf{d}_i^B = \mathbf{0} \quad \forall i\in I_A, \qquad
\mathbf{d}_i^A = \mathbf{0} \quad \forall i\in I_B.
\]
This blocks gradient flow from model \(B\)’s reconstruction loss into \(A\)-exclusive features, and vice versa. Shared features remain available to both branches. The intended decomposition is therefore explicit: common variance should accumulate in \(I_S\), while model-unique variance should accumulate in \(I_A\) or \(I_B\) [2602.11729].

Sparsity in the cross-architecture DFC is enforced with BatchTopK rather than an explicit \(L_1\) penalty. The reported target is \(k=200\), with an auxiliary-loss coefficient \(\alpha = 0.03\). Important tradeoff parameters include the exclusive partition size, described with examples such as 1%, 3%, and 5% of the dictionary for each model, and a sparsity annealing schedule from \(k_{\text{initial}}=1000\) down to \(200\) over 5000 steps in one setup [2602.11729].

The RL tool-use study preserves the same three-way conceptual partition but describes the implementation in terms of “A-exclusive, B-exclusive, and shared sub-dictionaries with gradient masking in order to enforce exclusivity.” There the two models are a base `Qwen/Qwen2.5-3B` and an RL-finetuned `chengq9/ToolRL-Qwen2.5-3B`, and the DFC is trained on concatenated residual-stream post-MLP activations \(h=(h_A,h_B)\) with top-\(k\) sparsity in the encoder [2606.26474]. This suggests that the central DFC principle is stable across settings even when the exact training loss and application differ.

## 3. Training protocol, alignment, and implementation regimes

For cross-architecture diffing, DFCs are trained on residual-stream activations from middle layers of paired models. Two main real-model comparisons are reported: `Llama-3.1-8B-Instruct` versus `Qwen3-8B`, and `GPT-OSS-20B` versus `Deepseek-R1-0528-Qwen3-8B`; for the latter, the paper specifies layer 12 of GPT-OSS-20B and layer 16 of Deepseek-R1-0528-Qwen3-8B, with activation dimensions \([4096, 2880]\) [2602.11729].

Because tokenizers differ, the paper aligns activations with a semantic window expansion algorithm: it steps through both token sequences, attempts one-to-one decoded-span matches, greedily expands the shorter side on mismatch, and when decoded text matches, retains the activation from the final token of each matched window. Reported alignment success rates are 99.2% for Llama/Qwen and 99.1% for GPT-OSS/Deepseek, with failures concentrated in special characters, non-standard quotes, emojis, and some multilingual or chat-formatting cases [2602.11729].

The training corpus for the cross-architecture DFC uses 100 million token-aligned activation pairs drawn from a 50/50 mixture of FineWeb and LMSYS-Chat-1M. Activations are normalized by scaling their median \(L_2\) norm to
\[
\sqrt{\frac{d_1 + d_2}{2}},
\]
and dynamic masking removes any activation whose norm is more than \(2\times\) the batch median from the loss for that step. The real-model setup uses dictionary expansion factor 32, total dictionary size \(131{,}072\), target sparsity \(k=200\), Adam with learning rate \(10^{-4}\), batch size \(2048\), 100,000 training steps, 1000-step warmup, bf16 precision, gradient checkpointing, and decoder norm scale initialization at 0.4 [2602.11729].

The RL tool-use study uses a same-architecture pair rather than a cross-architecture pair: both models are 36-layer Qwen2.5-3B systems with hidden dimension \(d=2048\). Its DFCs are trained on 40,000 FineWeb samples and 40,000 ToolRL instruction-output pairs. The paper reports a 48-crosscoder sweep comprising 36 DFCs and 12 CrossCoders, varying architecture, dictionary size \(D \in \{8192, 16384\}\), top-\(k \in \{45,90,160\}\), exclusive share \(p \in \{3\%,5\%,10\%\}\) for DFCs, and \(\lambda_{\text{excl}} \in \{0,10^{-3}\}\), with fixed settings of 9,000 steps, batch size 1,024, Adam with \(lr=10^{-4}\), and \(\lambda_{\text{sh}}=10^{-3}\) [2606.26474].

An implementation detail of particular interpretive importance is that the RL paper explicitly characterizes the A-exclusive partition not as a perfect sink for RL-specific capability, but as a filter that concentrates the most model-specific residue. That claim is tied to experiments showing that penalizing exclusive features with \(\lambda_{\text{excl}}=10^{-3}\) reduces Model A fidelity, rather than leaving performance unchanged [2606.26474].

## 4. Empirical findings and canonical case studies

The original cross-architecture DFC paper presents both toy-model and real-model evidence. In a synthetic setting with 2048 total concepts and 2.5% exclusive concepts per model, DFCs recovered more true exclusive concepts than standard crosscoders and Designated Shared Feature crosscoders, especially in the undercomplete regime, though at the cost of higher false-positive rates. The authors explicitly frame that tradeoff as acceptable for safety auditing, where missing a dangerous feature is worse than surfacing extra candidates [2602.11729].

In real-model experiments, the paper reports that DFCs surfaced interpretable model-exclusive features including CCP alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama-3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. The highlighted ideological and safety features all achieved maximum exclusivity score 5 under the paper’s transfer-based metric, and steering evaluations over 30 curated prompts showed strong causal control in the source model with near-zero effect in the comparison model [2602.11729].

The cross-architecture results are notable because standard quality metrics did not degrade. The reported comparison between DFC and standard crosscoder is essentially tied on fraction of variance explained, dead feature rate, and detection score:
- FVE: 0.817 vs 0.817
- dead features: 5.0% vs 5.6%
- detection score: 87.78% vs 87.77%  
The claimed gain is therefore selective rather than global: DFC improves exclusive-feature separation without harming normal crosscoder quality metrics [2602.11729].

The RL tool-use study supplies a second canonical case. Across its 48-crosscoder sweep, encode-decode reconstruction improves the RL model’s tool correctness by
\[
+31.1 \pm 9.7 \text{ pp},
\]
raising mean tool correctness from 19% to 50.1%. The same joint dictionary also produces passive transfer into the frozen base model, increasing its mean tool correctness by
\[
+6.8 \pm 5.0 \text{ pp},
\]
a phenomenon the authors call capability spillover. At the same time, format accuracy never spills over: \(\Delta_B\) format \(=0\) pp for all 48 runs [2606.26474].

The paper’s strongest mechanistic result is localization to one feature. In the flagship setting—`dfc-D8k-excl10-k45`, layer 13—steering one A-exclusive feature, identified as feature \(\#136\), yields
\[
+65.0 \text{ pp } \Delta \text{tool-correctness}
\]
with 95% CI \([+47.9,+82.1]\). This matches the performance of steering all 7 available A-exclusive neurons in that setup. By contrast, the CrossCoder reaches a higher unbudgeted best of \(+70.0\) pp only at \(|S|=33\), whereas the DFC A-exclusive partition saturates at \(|S|=1\) [2606.26474].

The same study reports a cross-layer mean best-cell improvement of
\[
\Delta = +43.3 \text{ pp}
\]
with 95% CI \([+22.7,+64.0]\), Cohen’s \(d_z=1.61\), paired \(t\) \(p=0.0006\), and Wilcoxon \(p=0.0078\), which the paper interprets as evidence that single-neuron saturation is not a layer-13 artefact [2606.26474]. Its mechanistic interpretation is correspondingly narrow: the dominant A-exclusive features appear to be structural-template detectors for `<tool_call>`, `<response>`, parameter boilerplate, and related Tool Interaction patterns, rather than a generic reasoning module.

## 5. Methodological relatives, critiques, and adjacent variants

DFCs emerged within a broader family of crosscoder methods, and several neighboring papers sharpen both their rationale and their limits. A central precursor is the observation that standard \(L_1\)-trained crosscoders can produce false model-only latents. “Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning” identifies two such failure modes—Complete Shrinkage and Latent Decoupling—and introduces Latent Scaling to diagnose them. The paper argues that a latent with no base decoder direction is not automatically a dedicated fine-tune-specific feature, and reports that the standard L1 crosscoder on Gemma 2 2B suffers heavily from these issues; BatchTopK substantially mitigates them and yields cleaner chat-only latents such as false-information detection, personal-question handling, and multiple refusal-related mechanisms [2504.02922].

Delta-Crosscoder occupies a closely related design space for narrow fine-tuning. It partitions the latent code into \(z=[z^{\text{shared}}, z_\Delta]\), uses a dedicated non-shared subspace for fine-tuning-induced variation, masks shared latents out of delta prediction, and adds an explicit delta loss
\[
\mathcal{L} = \mathcal{L}_{\text{recon}} + \lambda_s\,\text{sparsity}(z) + \lambda_\Delta \mathcal{L}_\Delta,
\]
with \(\lambda_\Delta = 0.005\). Evaluated across 10 model organisms, it achieves 10/10 causal recovery coverage, compared with 6/10 for DSF and 4/10 for two BatchTopK SAE baselines [2603.04426]. The paper explicitly reads this as a setting in which ordinary joint reconstruction underweights small, sparse, behaviorally decisive changes. This suggests that DFC-like dedication can be enforced not only by structural partitioning, but also by dedicated objectives and paired contrastive data.

A sharper critique comes from backdoor isolation. “Activation Differences Reveal Backdoors” compares an L1 Crosscoder on concatenated base and fine-tuned activations with a Diff-SAE trained directly on \(\Delta \mathbf{a} = \mathbf{a}_{\text{ft}} - \mathbf{a}_{\text{base}}\). At layer 18, Crosscoder achieves BIS \(0.010 \pm 0.011\) in the LoRA regime and \(0.000 \pm 0.000\) in the full-rank regime, while Diff-SAE achieves BIS \(0.400 \pm 0.025\) and \(0.400 \pm 0.026\) respectively [2605.07324]. The paper interprets the backdoor as a directional activation shift rather than a sparse feature that cleanly emerges in a joint latent basis. A plausible implication is that DFC-style partitioning is not universally sufficient; in some safety settings, the geometry of the signal may favor difference-first representations.

Beyond pairwise model diffing, several papers generalize the crosscoder paradigm in ways that bear on DFC. Cross-snapshot work on pretraining dynamics learns a shared sparse latent space with snapshot-specific decoders, using decoder norms to track when features emerge, persist, or split across checkpoints [2509.17196]. “Crosscoding Through Time” adds Relative Indirect Effects (RelIE) to identify which temporally aligned features are causally important for linguistic behavior [2509.05291]. “Sparse Crosscoders for diffing MoEs and Dense models” introduces explicitly designated shared features with tied decoders, though not a full three-way DFC partition; it reports about 87% fractional variance explained and finds that the MoE learns significantly fewer unique features than the dense model [2603.05805]. “Group Crosscoders for Mechanistic Analysis of Symmetry” shows that a crosscoder can be specialized around a transformation group, yielding feature families organized by symmetry rather than by model exclusivity [2410.24184]. Together these results place DFCs within a larger movement from generic joint sparse coding toward more structured latent decompositions.

## 6. Interpretation, limitations, and open questions

DFCs make a representation-level claim, not a capability-level claim. The original cross-architecture paper emphasizes that finding a model-exclusive feature does not mean the other model lacks the concept entirely; rather, one model has a particular sparse direction for that concept, and the other lacks a direct linear analogue in the learned aligned space [2602.11729]. This point is methodologically important because it limits how far exclusivity can be read as evidence about provenance or genuine novelty.

False positives remain an acknowledged tradeoff. The original DFC paper notes that many exclusive features do not correspond to meaningful model differences and treats DFC as a high-recall prescreening tool rather than a final proof of behavioral difference [2602.11729]. The RL paper reaches a similar conclusion from another angle: DFC does not perfectly isolate RL-induced tool-use capability, because tool-relevant information still leaks into shared features, causing capability spillover into the frozen base model. That paper therefore characterizes the exclusive partition as a filter rather than a sink [2606.26474].

Sensitivity to partition size, seed, and training stochasticity is another recurrent issue. In the cross-architecture paper, broad CCP-alignment-like features were robust, whereas American exceptionalism and finer-grained pro-China subfeatures were less consistently found [2602.11729]. In the RL paper, the strongest single-feature effect appears in a particular model-layer-configuration combination, even though cross-layer evidence suggests the phenomenon is broader [2606.26474]. This suggests that DFCs offer a useful inductive bias, but not a uniquely identifiable decomposition.

A final open question concerns when architectural dedication is the right inductive bias at all. The backdoor comparison with Diff-SAE indicates that if the behavior of interest is best modeled as a low-energy displacement vector in activation space, then even an exclusivity-aware crosscoder may be solving the wrong problem unless it builds the differential structure directly into the representation [2605.07324]. Conversely, the success of cross-architecture DFCs, Delta-Crosscoder, and the RL tool-use study indicates that when the target differences are sparse, semantically coherent, and at least partly localizable, explicit dedicated capacity can materially improve feature recovery [2602.11729].

Taken together, the literature defines DFCs as a structured crosscoder family whose distinctive contribution is to make exclusivity architectural rather than heuristic. Their main empirical use has been unsupervised model diffing across architectures and post-training regimes, especially where the goal is to surface model-specific ideological, safety, persona, or agentic mechanisms. Their main unresolved problem is not whether dedicated features can exist, but how reliably such features correspond to genuine differences in internal organization rather than artifacts of sparsity, alignment, or representation geometry.

Source: https://www.emergentmind.com/topics/dedicated-feature-crosscoders-dfc