---
title: BatchTopK Crosscoders
url: https://www.emergentmind.com/topics/batchtopk-crosscoders
type: topic
---

# BatchTopK Crosscoders

BatchTopK crosscoders are sparse dictionary-learning models for joint analysis of paired activation spaces, typically from two models evaluated on the same input. They extend the crosscoder formulation by replacing soft sparsity regularization with a hard batchwise sparsity budget, so that a single latent feature space reconstructs both activation vectors while a fixed number of latent activations are retained across a batch. In current mechanistic-interpretability practice, they are used primarily for model diffing: comparing dense and Mixture-of-Experts internals, base and chat-fine-tuned models, narrow fine-tuning regimes, and cross-architecture model pairs. Reported motivations are improved control over latent competition, reduced sparsity artifacts relative to \(L_1\)-style objectives, and better isolation of shared versus model-specific structure, though recent work also emphasizes that feature geometry, latent competition, and evaluation methodology remain important limitations [2603.05805].

## 1. Formal definition and relation to other sparse dictionary learners

A crosscoder is a sparse-autoencoder-like model trained on paired activations from two systems. Rather than compressing one activation space, it learns a single set of latent features \(f_i(x)\) that reconstructs both activation vectors \(x^A\) and \(x^B\) from the same input token. In the notation used for dense-versus-MoE comparison, the decoder reconstructs both models as
\[
\hat{x}^m = \sum_{i=1}^k f_i(x)\, W^m_{\text{dec}, i}
\quad \text{for } m \in \{A, B\}.
\]
Here each feature index \(i\) has one latent activation but model-specific decoder vectors. If a latent reconstructs both models similarly, it is interpreted as shared; if its decoder norm is much larger in one model than the other, it is interpreted as model-specific [2603.05805].

This differs from a standard sparse autoencoder, which reconstructs a single activation space. It also differs from a transcoder, which maps between layers rather than jointly reconstructing multiple representations. In the unified sparse dictionary learning framework, crosscoders are the multi-source case: inputs and targets are concatenations across representations, and the latent space is intended to discover shared features across those sources [2512.05534].

Not every crosscoder in the recent literature uses BatchTopK. Cross-snapshot crosscoders for tracking feature evolution across pre-training checkpoints, for example, use JumpReLU thresholding together with a tanh-based differentiable \(L_0\) approximation and a quadratic frequency penalty; that work explicitly does not use BatchTopK, top-\(k\) batching, or a batching approximation to top-\(k\) selection [2509.17196]. Conversely, not every BatchTopK dictionary learner is a crosscoder: in Vision Transformers, BatchTopK sparse autoencoders have been used only as single-layer probes on block-11 residual-stream activations, not as inter-representation alignment modules [2603.15919].

## 2. BatchTopK as the sparsity mechanism

In standard crosscoder training, sparsity can be encouraged by a soft regularizer. For paired dense and MoE activations, the reported objective is
\[
\mathcal{L} = \mathbb{E}_x \Bigg[ \lVert x^A - \hat{x}^A \rVert^2 + \lVert x^B - \hat{x}^B \rVert^2 + \lambda \sum_{i=1}^k f_i(x)\bigl(\lVert W^A_{\text{dec}, i} \rVert + \lVert W^B_{\text{dec}, i} \rVert \bigr) \Bigg].
\]
The reported concern is that this continuous sparsity objective can overestimate shared structure when the compared activation spaces are structurally quite different. In the dense-versus-MoE setting, a standard crosscoder trained on the two models “classified” many features as shared even when the decoder vectors had near-zero cosine similarity [2603.05805].

BatchTopK replaces that soft penalty with a hard sparsity budget. Over a batch, it keeps only the top activations
\[
\operatorname{TopK}\Bigl(f_i(x)\, \bigl(\lVert W^A_{\text{dec},i}\rVert + \lVert W^B_{\text{dec},i}\rVert \bigr)\Bigr)
\]
and zeros out the rest. Sparsity is therefore enforced by selecting a fixed number of active latents rather than merely discouraging large activations through an \(L_1\)-style penalty [2603.05805].

A closely related motivation appears in base-versus-chat model diffing. There, the standard \(L_1\) crosscoder objective is reported to produce two artifacts: **Complete Shrinkage**, in which one decoder norm can collapse toward zero even when the latent remains useful for that model, and **Latent Decoupling**, in which a shared concept splits into separate model-specific latents. The BatchTopK crosscoder replaces the \(L_1\) penalty with batchwise top-\(k\) selection and is reported to substantially mitigate both artifacts [2504.02922].

BatchTopK itself is a batch-level generalization of TopK sparsification. Standard TopK SAE keeps each token’s top \(K\) latents independently; BatchTopK instead selects activations across an entire batch, allowing some tokens more than \(K\) active latents and others fewer while preserving the average sparsity budget. This improves average reconstruction, but later work identifies an “activation lottery” in which rare high-magnitude activations repeatedly win sparse slots and crowd out more informative but lower-magnitude features [2508.21324].

## 3. Shared, exclusive, and delta-reserved latent structure

A major design question in BatchTopK crosscoders is whether shared and exclusive capacity are imposed only post hoc or baked into the architecture. One approach is the **fixed shared-feature** crosscoder. In this variant, a subset \(S\) of feature indices is designated as shared, the remaining features \(F = \{1,\dots,k\}\setminus S\) are exclusive, shared features have tied decoder parameters across models, and separate sparsity penalties are used for shared and exclusive features. In the dense-versus-MoE study, prior work suggested \(\lambda_s/\lambda_f \approx 0.1\text{–}0.2\), but the reported comparison required a much larger ratio of about \(0.7\) to obtain good reconstructions and meaningful separation of shared versus exclusive features, because the dense and MoE models were trained from scratch rather than being a base/fine-tuned pair [2603.05805].

A stronger exclusivity construction is the **Dedicated Feature Crosscoder** (DFC). DFC partitions the dictionary into three disjoint sets: features exclusive to model \(A\), features exclusive to model \(B\), and shared features. The decoder weights are structurally zeroed across partitions, so an \(A\)-exclusive feature has no decoder contribution to model \(B\), and exclusive features receive gradient only from their own model’s reconstruction. In reported cross-architecture experiments, this architectural partitioning is the main novelty, while BatchTopK is the sparsity mechanism used to train both standard crosscoders and DFCs [2602.11729].

A further specialization appears in **Delta-Crosscoder**, designed for narrow fine-tuning regimes. There the latent code is split into shared and non-shared components,
\[
z = [z_{\text{shared}}, z_{\Delta}],
\]
with **20% shared** and **80% reserved for non-shared/delta features**. Sparsity is enforced with a **Dual-K BatchTopK** scheme in which shared latents receive a larger budget and non-shared latents a smaller budget, and shared features are masked out of the delta term so they cannot absorb fine-tuning-specific differences [2603.04426].

| Design | Structural rule | Reported purpose |
|---|---|---|
| Fixed shared-feature crosscoder | Reserve subset \(S\) as shared; tie shared decoders | Disentangle common from model-specific structure |
| DFC | Partition into \(I_A\), \(I_B\), \(I_S\); structurally zero cross-partition decoders | Enforce exclusivity by construction |
| Delta-Crosscoder | Split \(z\) into \(z_{\text{shared}}\) and \(z_{\Delta}\); apply Dual-K BatchTopK | Reserve capacity for fine-tuning-specific changes |

These variants show that BatchTopK crosscoders are not a single architecture but a family of joint sparse dictionary learners whose main commonality is batchwise sparse latent selection over paired activations.

## 4. Empirical uses and reported findings

The most detailed dense-versus-MoE application trains a BatchTopK crosscoder on **the third layer activations** of a **5-layer** dense transformer and a **5-layer** MoE transformer trained on roughly **1 billion tokens** total, with equal thirds from ArXiv / RedPajama scientific text, StarCoder code, and SimpleStories English stories. Each upstream model is trained for **2 epochs** using cross-entropy loss, and the MoE additionally uses a **Switch-style load-balancing loss**. The models are matched for **equal active parameters**. In that setting, the best crosscoder—the BatchTopK version with explicit shared features—achieves about **87% fractional variance explained** over **40K training steps**. It reports **18,940 shared features** and **3,226 dense-only features**, with the MoE-specific feature count stated to be significantly smaller than the dense-specific count. The reported interpretive conclusion is that the MoE learns fewer unique features, and that MoE-only features have higher activation density than shared features whereas dense-only features have lower density [2603.05805].

In chat-tuning analysis, BatchTopK crosscoders are trained on **Gemma 2 2B base** and **Gemma 2 2B-it**, at **layer 13 of 26**, with **expansion factor 32** and about **73,728 latents**. Training data come from **FineWeb** and **LMSYS-chat-1M**. The reported target sparsity is matched across objectives: the \(L_1\) crosscoder is tuned to validation \(L_0 \approx 101.0\), and the BatchTopK crosscoder uses \(k=100\) with reported validation \(L_0 \approx 99.48\). BatchTopK is reported to find more genuinely chat-specific and highly interpretable latents, including **false information**, **personal question**, and several refusal-related latents with different refusal triggers [2504.02922].

In narrow fine-tuning regimes, Delta-Crosscoder combines BatchTopK sparsity with a delta-based loss
\[
L_{\Delta} = \left\lVert \Delta - (W_{\text{ft}} - W_{\text{base}})z \right\rVert_2^2
\]
and contrastive paired activations from matched prompts and responses. Evaluated across **10 model organisms**, including synthetic false facts, emergent misalignment, subliminal learning, and taboo word guessing, on Gemma, LLaMA, and Qwen models from **1B-9B parameters**, it is reported to cover **all 10/10 organisms**, whereas DSF succeeds on **6/10**, and BatchTopK-200 and BatchTopK-400 each on **4/10**. Reconstruction quality remains within about **1–2% absolute** of standard crosscoder baselines, typically around **80%** explained variance in the LLaMA emergent-misalignment settings, and some settings show **more than 2x fewer dead features** than BatchTopK-400 [2603.04426].

In cross-architecture model diffing, BatchTopK-trained crosscoders and DFCs are applied to **Llama-3.1-8B-Instruct vs Qwen3-8B** and **GPT-OSS-20B vs Deepseek-R1-0528-Qwen3-8B**. The reported setup uses **100 million token-aligned activation pairs per diff**, a **50/50 mix of FineWeb and LMSYS-Chat-1M**, **dictionary size 131,072**, **expansion factor 32**, target sparsity **\(k=200\)**, **Adam**, learning rate \(10^{-4}\), **batch size 2048**, **100,000 training steps**, **1,000** warmup steps, decoder norm scale **0.4**, and **bf16** mixed precision. Reported exclusive discoveries include **Chinese Communist Party alignment** in Qwen and Deepseek-R1-0528-Qwen3-8B, **American exceptionalism** in Llama-3.1-8B-Instruct, and a **copyright refusal mechanism** in GPT-OSS-20B [2602.11729].

## 5. Evaluation, feature classification, and geometric caveats

BatchTopK crosscoder analyses usually classify features by decoder norms, decoder-direction alignment, and activation statistics. In the dense-versus-MoE comparison, the relative decoder-norm difference \(\Delta_{\mathrm{norm}(i)}\) is interpreted so that values near \(0\) indicate MoE-specific features, values near \(1\) indicate dense-specific features, and values near \(0.5\) indicate shared features. That same study emphasizes that norm-based “sharedness” is not sufficient for semantic alignment: some explicitly budgeted shared features have cosine similarity near \(1\), but many features labeled “shared” by intermediate \(\Delta_{\text{norm}}\) values in the range \(0.3\) to \(0.7\) do not have strongly aligned decoder vectors, and some have cosine similarity near \(-1\). The authors also report that they do **not** observe a clean trimodal structure in feature similarity or density comparable to the classic base-versus-finetuned crosscoder setting [2603.05805].

In base-versus-chat diffing, evaluation includes **Latent Scaling**, which fits scalar coefficients for a latent’s contribution to reconstruction or error and derives two ratios, \(\nu_j^r\) and \(\nu_j^\varepsilon\). These are used to distinguish genuinely chat-specific latents from artifacts caused by Complete Shrinkage or Latent Decoupling. Reported results show that many \(L_1\)-identified “chat-only” latents overlap strongly with shared-latent behavior, whereas the BatchTopK chat-only set exhibits much less overlap and near-zero \(\nu^\varepsilon\) [2504.02922].

Cross-architecture diffing adds transfer-based evaluation. The reported **exclusivity score** is defined as \(6 - \text{Similarity Score}\), where similarity is assessed using transferred steering vectors and LLM judgment. The authors emphasize that cross-architecture diffing is **high-recall, not high-precision**, and that exclusivity is about representation rather than the underlying concept, so many apparently exclusive features may still correspond to generic safety concepts or duplicated behaviors represented differently across models [2602.11729].

These caveats indicate that BatchTopK crosscoder outputs usually require both quantitative and qualitative analysis. Decoder norms, cosine similarity, activation density, latent scaling, and causal interventions measure different aspects of the learned feature geometry and can disagree.

## 6. Limitations, extensions, and broader interpretability context

Several limitations recur across the literature. First, batch-level latent competition is not automatically benign. Distribution-aware feature-selection work argues that naïve BatchTopK can suffer an **activation lottery**, in which rare high-magnitude features crowd out more stable mid-frequency features. That work introduces Sampled-SAE as a two-stage alternative, and although it does not train crosscoders directly, it explicitly frames the result as relevant to BatchTopK-style crosscoders because they also rely on sparse, shared latent feature selection across many token activations. A plausible implication is that batch competition in crosscoders may also need to be distribution-aware rather than fully open [2508.21324].

Second, recent work on cosine-scored sparse autoencoders argues that standard inner-product encoders have the wrong geometry for normalized transformer representations under BatchTopK, because high-norm tokens inflate all pre-activations and claim dictionary slots regardless of content alignment. That paper studies SAEs rather than crosscoders, but it explicitly presents the result as relevant to BatchTopK Crosscoders and other dictionary-learning methods on normalized activations. A plausible implication is that BatchTopK crosscoders trained on normalized residual streams may also waste capacity on norm-sensitive features unless encoder scoring is made more cosine-like [2606.15054].

Third, the current theoretical account remains broad rather than BatchTopK-crosscoder-specific. The unified sparse dictionary learning theory treats crosscoders as one SDL instantiation and describes Batch Top-\(k\) as a batchwise activation rule that allows different samples to have different numbers of activated features, but it does **not** provide a dedicated theorem for BatchTopK crosscoders. Its main theoretical message is instead that sparse dictionary learning can contain spurious local minima explaining **feature absorption**, **dead neurons**, and the motivation for resampling [2512.05534].

Finally, recent empirical work is explicit that crosscoder adaptation is still incomplete for structurally distinct architectures. In dense-versus-MoE diffing, the authors state that crosscoders need further adaptation to fully capture differences between dense models and MoEs, and that qualitative feature inspection remains an important next step [2603.05805]. This suggests that BatchTopK crosscoders are best regarded as a constrained joint dictionary-learning method whose usefulness depends not only on reconstruction quality and sparsity, but also on the architectural regime, feature-selection policy, and the geometry of the underlying activation space.

Source: https://www.emergentmind.com/topics/batchtopk-crosscoders