BatchTopK Crosscoders

• BatchTopK crosscoders are sparse dictionary-learning models that employ a hard batchwise sparsity budget to jointly reconstruct paired activation vectors.
• They enable model diffing by isolating shared versus model-specific latent features, reducing artifacts common to L1-style regularization.
• Design variants like fixed shared-feature, Dedicated Feature Crosscoder, and Delta-Crosscoder illustrate practical applications in dense-versus-MoE comparisons and fine-tuning regimes.

BatchTopK crosscoders are sparse dictionary-learning models for joint analysis of paired activation spaces, typically from two models evaluated on the same input. They extend the crosscoder formulation by replacing soft sparsity regularization with a hard batchwise sparsity budget, so that a single latent feature space reconstructs both activation vectors while a fixed number of latent activations are retained across a batch. In current mechanistic-interpretability practice, they are used primarily for model diffing: comparing dense and Mixture-of-Experts internals, base and chat-fine-tuned models, narrow fine-tuning regimes, and cross-architecture model pairs. Reported motivations are improved control over latent competition, reduced sparsity artifacts relative to $L_1$-style objectives, and better isolation of shared versus model-specific structure, though recent work also emphasizes that feature geometry, latent competition, and evaluation methodology remain important limitations (Chaudhari et al., 6 Mar 2026).

1. Formal definition and relation to other sparse dictionary learners

A crosscoder is a sparse-autoencoder-like model trained on paired activations from two systems. Rather than compressing one activation space, it learns a single set of latent features $f_i(x)$ that reconstructs both activation vectors $x^A$ and $x^B$ from the same input token. In the notation used for dense-versus-MoE comparison, the decoder reconstructs both models as

$$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$

Here each feature index $i$ has one latent activation but model-specific decoder vectors. If a latent reconstructs both models similarly, it is interpreted as shared; if its decoder norm is much larger in one model than the other, it is interpreted as model-specific (Chaudhari et al., 6 Mar 2026).

This differs from a standard sparse autoencoder, which reconstructs a single activation space. It also differs from a transcoder, which maps between layers rather than jointly reconstructing multiple representations. In the unified sparse dictionary learning framework, crosscoders are the multi-source case: inputs and targets are concatenations across representations, and the latent space is intended to discover shared features across those sources (Tang et al., 5 Dec 2025).

Not every crosscoder in the recent literature uses BatchTopK. Cross-snapshot crosscoders for tracking feature evolution across pre-training checkpoints, for example, use JumpReLU thresholding together with a tanh-based differentiable $L_0$ approximation and a quadratic frequency penalty; that work explicitly does not use BatchTopK, top-$k$ batching, or a batching approximation to top-$k$ selection (Ge et al., 21 Sep 2025). Conversely, not every BatchTopK dictionary learner is a crosscoder: in Vision Transformers, BatchTopK sparse autoencoders have been used only as single-layer probes on block-11 residual-stream activations, not as inter-representation alignment modules (Zhang, 16 Mar 2026).

2. BatchTopK as the sparsity mechanism

In standard crosscoder training, sparsity can be encouraged by a soft regularizer. For paired dense and MoE activations, the reported objective is

$$\mathcal{L} = \mathbb{E}_x \Bigg[ \lVert x^A - \hat{x}^A \rVert^2 + \lVert x^B - \hat{x}^B \rVert^2 + \lambda \sum_{i=1}^k f_i(x)\bigl(\lVert W^A_{\text{dec}, i} \rVert + \lVert W^B_{\text{dec}, i} \rVert \bigr) \Bigg].$$

The reported concern is that this continuous sparsity objective can overestimate shared structure when the compared activation spaces are structurally quite different. In the dense-versus-MoE setting, a standard crosscoder trained on the two models “classified” many features as shared even when the decoder vectors had near-zero cosine similarity (Chaudhari et al., 6 Mar 2026).

BatchTopK replaces that soft penalty with a hard sparsity budget. Over a batch, it keeps only the top activations

$f_i(x)$0

and zeros out the rest. Sparsity is therefore enforced by selecting a fixed number of active latents rather than merely discouraging large activations through an $f_i(x)$1-style penalty (Chaudhari et al., 6 Mar 2026).

A closely related motivation appears in base-versus-chat model diffing. There, the standard $f_i(x)$2 crosscoder objective is reported to produce two artifacts: Complete Shrinkage, in which one decoder norm can collapse toward zero even when the latent remains useful for that model, and Latent Decoupling, in which a shared concept splits into separate model-specific latents. The BatchTopK crosscoder replaces the $f_i(x)$3 penalty with batchwise top-$f_i(x)$4 selection and is reported to substantially mitigate both artifacts (Minder et al., 3 Apr 2025).

BatchTopK itself is a batch-level generalization of TopK sparsification. Standard TopK SAE keeps each token’s top $f_i(x)$5 latents independently; BatchTopK instead selects activations across an entire batch, allowing some tokens more than $f_i(x)$6 active latents and others fewer while preserving the average sparsity budget. This improves average reconstruction, but later work identifies an “activation lottery” in which rare high-magnitude activations repeatedly win sparse slots and crowd out more informative but lower-magnitude features (Oozeer et al., 29 Aug 2025).

3. Shared, exclusive, and delta-reserved latent structure

A major design question in BatchTopK crosscoders is whether shared and exclusive capacity are imposed only post hoc or baked into the architecture. One approach is the fixed shared-feature crosscoder. In this variant, a subset $f_i(x)$7 of feature indices is designated as shared, the remaining features $f_i(x)$8 are exclusive, shared features have tied decoder parameters across models, and separate sparsity penalties are used for shared and exclusive features. In the dense-versus-MoE study, prior work suggested $f_i(x)$9, but the reported comparison required a much larger ratio of about $x^A$0 to obtain good reconstructions and meaningful separation of shared versus exclusive features, because the dense and MoE models were trained from scratch rather than being a base/fine-tuned pair (Chaudhari et al., 6 Mar 2026).

A stronger exclusivity construction is the Dedicated Feature Crosscoder (DFC). DFC partitions the dictionary into three disjoint sets: features exclusive to model $x^A$1, features exclusive to model $x^A$2, and shared features. The decoder weights are structurally zeroed across partitions, so an $x^A$3-exclusive feature has no decoder contribution to model $x^A$4, and exclusive features receive gradient only from their own model’s reconstruction. In reported cross-architecture experiments, this architectural partitioning is the main novelty, while BatchTopK is the sparsity mechanism used to train both standard crosscoders and DFCs (Jiralerspong et al., 12 Feb 2026).

A further specialization appears in Delta-Crosscoder, designed for narrow fine-tuning regimes. There the latent code is split into shared and non-shared components,

$x^A$5

with 20% shared and 80% reserved for non-shared/delta features. Sparsity is enforced with a Dual-K BatchTopK scheme in which shared latents receive a larger budget and non-shared latents a smaller budget, and shared features are masked out of the delta term so they cannot absorb fine-tuning-specific differences (Kassem et al., 16 Feb 2026).

Design	Structural rule	Reported purpose
Fixed shared-feature crosscoder	Reserve subset $x^A$6 as shared; tie shared decoders	Disentangle common from model-specific structure
DFC	Partition into $x^A$7, $x^A$8, $x^A$9; structurally zero cross-partition decoders	Enforce exclusivity by construction
Delta-Crosscoder	Split $x^B$0 into $x^B$1 and $x^B$2; apply Dual-K BatchTopK	Reserve capacity for fine-tuning-specific changes

These variants show that BatchTopK crosscoders are not a single architecture but a family of joint sparse dictionary learners whose main commonality is batchwise sparse latent selection over paired activations.

4. Empirical uses and reported findings

The most detailed dense-versus-MoE application trains a BatchTopK crosscoder on the third layer activations of a 5-layer dense transformer and a 5-layer MoE transformer trained on roughly 1 billion tokens total, with equal thirds from ArXiv / RedPajama scientific text, StarCoder code, and SimpleStories English stories. Each upstream model is trained for 2 epochs using cross-entropy loss, and the MoE additionally uses a Switch-style load-balancing loss. The models are matched for equal active parameters. In that setting, the best crosscoder—the BatchTopK version with explicit shared features—achieves about 87% fractional variance explained over 40K training steps. It reports 18,940 shared features and 3,226 dense-only features, with the MoE-specific feature count stated to be significantly smaller than the dense-specific count. The reported interpretive conclusion is that the MoE learns fewer unique features, and that MoE-only features have higher activation density than shared features whereas dense-only features have lower density (Chaudhari et al., 6 Mar 2026).

In chat-tuning analysis, BatchTopK crosscoders are trained on Gemma 2 2B base and Gemma 2 2B-it, at layer 13 of 26, with expansion factor 32 and about 73,728 latents. Training data come from FineWeb and LMSYS-chat-1M. The reported target sparsity is matched across objectives: the $x^B$3 crosscoder is tuned to validation $x^B$4, and the BatchTopK crosscoder uses $x^B$5 with reported validation $x^B$6. BatchTopK is reported to find more genuinely chat-specific and highly interpretable latents, including false information, personal question, and several refusal-related latents with different refusal triggers (Minder et al., 3 Apr 2025).

In narrow fine-tuning regimes, Delta-Crosscoder combines BatchTopK sparsity with a delta-based loss

$x^B$7

and contrastive paired activations from matched prompts and responses. Evaluated across 10 model organisms, including synthetic false facts, emergent misalignment, subliminal learning, and taboo word guessing, on Gemma, LLaMA, and Qwen models from 1B-9B parameters, it is reported to cover all 10/10 organisms, whereas DSF succeeds on 6/10, and BatchTopK-200 and BatchTopK-400 each on 4/10. Reconstruction quality remains within about 1–2% absolute of standard crosscoder baselines, typically around 80% explained variance in the LLaMA emergent-misalignment settings, and some settings show more than 2x fewer dead features than BatchTopK-400 (Kassem et al., 16 Feb 2026).

In cross-architecture model diffing, BatchTopK-trained crosscoders and DFCs are applied to Llama-3.1-8B-Instruct vs Qwen3-8B and GPT-OSS-20B vs Deepseek-R1-0528-Qwen3-8B. The reported setup uses 100 million token-aligned activation pairs per diff, a 50/50 mix of FineWeb and LMSYS-Chat-1M, dictionary size 131,072, expansion factor 32, target sparsity $x^B$8, Adam, learning rate $x^B$9, batch size 2048, 100,000 training steps, 1,000 warmup steps, decoder norm scale 0.4, and bf16 mixed precision. Reported exclusive discoveries include Chinese Communist Party alignment in Qwen and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama-3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B (Jiralerspong et al., 12 Feb 2026).

5. Evaluation, feature classification, and geometric caveats

BatchTopK crosscoder analyses usually classify features by decoder norms, decoder-direction alignment, and activation statistics. In the dense-versus-MoE comparison, the relative decoder-norm difference $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$0 is interpreted so that values near $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$1 indicate MoE-specific features, values near $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$2 indicate dense-specific features, and values near $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$3 indicate shared features. That same study emphasizes that norm-based “sharedness” is not sufficient for semantic alignment: some explicitly budgeted shared features have cosine similarity near $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$4, but many features labeled “shared” by intermediate $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$5 values in the range $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$6 to $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$7 do not have strongly aligned decoder vectors, and some have cosine similarity near $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$8. The authors also report that they do not observe a clean trimodal structure in feature similarity or density comparable to the classic base-versus-finetuned crosscoder setting (Chaudhari et al., 6 Mar 2026).

In base-versus-chat diffing, evaluation includes Latent Scaling, which fits scalar coefficients for a latent’s contribution to reconstruction or error and derives two ratios, $$\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i} \quad \text{for } m \in \{A, B\}.$$9 and $i$0. These are used to distinguish genuinely chat-specific latents from artifacts caused by Complete Shrinkage or Latent Decoupling. Reported results show that many $i$1-identified “chat-only” latents overlap strongly with shared-latent behavior, whereas the BatchTopK chat-only set exhibits much less overlap and near-zero $i$2 (Minder et al., 3 Apr 2025).

Cross-architecture diffing adds transfer-based evaluation. The reported exclusivity score is defined as $i$3, where similarity is assessed using transferred steering vectors and LLM judgment. The authors emphasize that cross-architecture diffing is high-recall, not high-precision, and that exclusivity is about representation rather than the underlying concept, so many apparently exclusive features may still correspond to generic safety concepts or duplicated behaviors represented differently across models (Jiralerspong et al., 12 Feb 2026).

These caveats indicate that BatchTopK crosscoder outputs usually require both quantitative and qualitative analysis. Decoder norms, cosine similarity, activation density, latent scaling, and causal interventions measure different aspects of the learned feature geometry and can disagree.

6. Limitations, extensions, and broader interpretability context

Several limitations recur across the literature. First, batch-level latent competition is not automatically benign. Distribution-aware feature-selection work argues that naïve BatchTopK can suffer an activation lottery, in which rare high-magnitude features crowd out more stable mid-frequency features. That work introduces Sampled-SAE as a two-stage alternative, and although it does not train crosscoders directly, it explicitly frames the result as relevant to BatchTopK-style crosscoders because they also rely on sparse, shared latent feature selection across many token activations. A plausible implication is that batch competition in crosscoders may also need to be distribution-aware rather than fully open (Oozeer et al., 29 Aug 2025).

Second, recent work on cosine-scored sparse autoencoders argues that standard inner-product encoders have the wrong geometry for normalized transformer representations under BatchTopK, because high-norm tokens inflate all pre-activations and claim dictionary slots regardless of content alignment. That paper studies SAEs rather than crosscoders, but it explicitly presents the result as relevant to BatchTopK Crosscoders and other dictionary-learning methods on normalized activations. A plausible implication is that BatchTopK crosscoders trained on normalized residual streams may also waste capacity on norm-sensitive features unless encoder scoring is made more cosine-like (Naihin et al., 13 Jun 2026).

Third, the current theoretical account remains broad rather than BatchTopK-crosscoder-specific. The unified sparse dictionary learning theory treats crosscoders as one SDL instantiation and describes Batch Top-$i$4 as a batchwise activation rule that allows different samples to have different numbers of activated features, but it does not provide a dedicated theorem for BatchTopK crosscoders. Its main theoretical message is instead that sparse dictionary learning can contain spurious local minima explaining feature absorption, dead neurons, and the motivation for resampling (Tang et al., 5 Dec 2025).

Finally, recent empirical work is explicit that crosscoder adaptation is still incomplete for structurally distinct architectures. In dense-versus-MoE diffing, the authors state that crosscoders need further adaptation to fully capture differences between dense models and MoEs, and that qualitative feature inspection remains an important next step (Chaudhari et al., 6 Mar 2026). This suggests that BatchTopK crosscoders are best regarded as a constrained joint dictionary-learning method whose usefulness depends not only on reconstruction quality and sparsity, but also on the architectural regime, feature-selection policy, and the geometry of the underlying activation space.