---
title: AdaptiveK Sparse Autoencoders
url: https://www.emergentmind.com/topics/adaptive-top-k-sparse-autoencoders-adaptivek
type: topic
---

# AdaptiveK Sparse Autoencoders

Adaptive Top K Sparse Autoencoders, commonly abbreviated **AdaptiveK**, are sparse autoencoders in which the latent support size is not a fixed global constant but a quantity that varies with the input. In contrast to standard TopK sparse autoencoders, which allocate exactly \(K\) active latents to every sample, AdaptiveK methods are motivated by the claim that real activations differ in semantic complexity and local intrinsic dimensionality, so the number of explanatory factors required for faithful and interpretable reconstruction should also vary across samples [2508.17320]. In current arXiv usage, the term covers both explicit per-input \(k\)-prediction mechanisms and closely related designs that achieve variable support size through batch-level competition or data-dependent sparse selection; among the clearest explicit instances are the complexity-driven **AdaptiveK Sparse Autoencoders** [2508.17320] and **SoftSAE**, which is described as “essentially a concrete AdaptiveK SAE instantiation” [2605.06610].

## 1. Fixed-\(K\) origins and the motivation for adaptation

The immediate background to AdaptiveK is the fixed-cardinality **k-sparse autoencoder**. In the original k-Sparse Autoencoder, the encoder computes
\[
\boldsymbol{z}=W^\top \boldsymbol{x}+\boldsymbol{b},
\]
then keeps only the \(k\) largest hidden activations and sets the rest to zero, yielding exact per-example sparsity with linear reconstruction and gradient flow only through the active units [1312.5663]. Modern TopK SAEs in mechanistic interpretability retain this basic hard-support idea, but apply it to overcomplete dictionaries trained on residual-stream or related activations [2406.04093].

The central objection to fixed \(K\) is that it imposes the same representational budget on all inputs. The AdaptiveK paper states that some contexts are “semantically simple” while others are “denser, more abstract, more domain-specific, or more logically entangled,” and therefore should not be forced to use the same number of SAE features [2508.17320]. SoftSAE makes the same point in geometric language: natural data lies on manifolds with varying local intrinsic dimensionality, so a fixed \(K\) gives simple examples noisy, overly long explanations and complex examples insufficient capacity [2605.06610].

This critique is also visible inside the large-scale fixed-TopK literature. “Scaling and evaluating sparse autoencoders” explicitly states that forcing every token to use exactly \(k\) latents is “likely suboptimal” and that, ideally, one would constrain \(\mathbb E[L_0]\) rather than \(L_0\) itself [2406.04093]. SplInterp reaches a related conclusion from a different direction: its optimal piecewise-affine baseline is a local-PCA-style autoencoder with region-specific rank \(K_i\), which suggests that variable local rank is theoretically natural even though the paper itself analyzes fixed-\(K\) TopK partitions [2505.11836].

## 2. Canonical AdaptiveK formulations

A standard SAE baseline in this literature uses
\[
z=\mathrm{ReLU}(W_{\mathrm{enc}}(x-b_{\mathrm{pre}})+b_{\mathrm{enc}}),\qquad
\hat{x}=W_{\mathrm{dec}}z+b_{\mathrm{pre}}.
\]
AdaptiveK augments this with a complexity estimator and replaces the fixed TopK budget by an input-dependent one [2508.17320].

In “AdaptiveK Sparse Autoencoders: Dynamic Sparsity Allocation for Interpretable LLM Representations,” the adaptive signal is a linear probe trained to predict a scalar complexity score from the activation:
\[
\hat{y}=\hat{w}^T x+\hat{b}.
\]
The probe is trained by ridge regression with objective
\[
L(w,b)=\frac{1}{n}\sum_{i=1}^{n}(y_i-(w^Tx_i+b))^2+\frac{\lambda}{2}\|w\|_2^2,
\]
and closed-form solution
\[
\hat{w}=(A^TA+\lambda I)^{-1}A^Ty.
\]
The predicted complexity \(c=\hat{w}^Tx+\hat{b}\) is then mapped by a sigmoid-based rule to an adaptive support size \(k_{\mathrm{adp}}\) bounded between \(k_{\min}\) and \(k_{\max}\), and TopK is applied with that budget [2508.17320]. The paper reports \(base\_k=80\), \(min\_k=20\), \(max\_k=320\), and sigmoid steepness \(0.6\).

SoftSAE implements the same general idea by making the support size itself a learned latent decision. It starts from
\[
z=\mathrm{ReLU}(W_{\mathrm{enc}}(x-b_{\mathrm{pre}})+b_{\mathrm{enc}}),\qquad
\hat{x}=W_{\mathrm{dec}}z+b_{\mathrm{pre}},
\]
then predicts a continuous budget
\[
\hat{k}:=\sigma(\mathrm{MLP}(x))\cdot k_{\max},
\]
with a single-hidden-layer Dynamic Sparsity MLP whose first layer is initialized from \(W_{\mathrm{enc}}\) [2605.06610]. During training it applies a differentiable soft top-\(k\) operator,
\[
p(z)=\mathrm{SoftTopK}(z,\hat{k},\alpha)\in(0,1)^d,\qquad \sum_i p_i=\hat{k},
\]
and uses
\[
f_{\mathrm{train}}(z)=p(z)\odot z.
\]
During inference, and in the late hardening phase of training, it switches to hard TopK with rounded \(\hat{k}\) [2605.06610].

The distinction between these two formulations is architectural rather than conceptual. AdaptiveK uses an externally pretrained complexity signal to set \(k\), whereas SoftSAE predicts \(\hat{k}(x)\) directly inside the SAE and trains the whole mechanism end to end. Both are explicit AdaptiveK designs in the sense that the model makes a sample-dependent decision about **how many** features should be active, not merely **which** ones.

## 3. The broader design space of variable-support SAEs

Not every variable-support sparse autoencoder is an explicit AdaptiveK model. The literature now contains several mechanisms that relax fixed TopK in different ways.

| Method | Adaptive mechanism | Relation to AdaptiveK |
|---|---|---|
| **AdaptiveK SAE** [2508.17320] | Linear-probe complexity prediction mapped to \(k_{\mathrm{adp}}\) | Explicit learned per-context budget |
| **SoftSAE** [2605.06610] | Dynamic Sparsity MLP predicts continuous \(\hat{k}(x)\) with differentiable SoftTopK | Explicit per-input AdaptiveK instantiation |
| **BatchTopK SAE** [2412.06410] | Top \(nk\) activations kept across a batch, so samplewise \(L_0\) varies while average stays fixed | Adaptive per sample only in a batch-average sense |
| **Sparsemax attention SAE** [2604.14925] | Sparsemax infers support size from a simplex projection over concept scores | AdaptiveK-like, but attention-based rather than TopK-with-\(k(x)\) |
| **Sampled-SAE** [2508.21324] | Batch-level feature scoring restricts the candidate pool before BatchTopK | Distribution-aware feature eligibility, not explicit tokenwise \(k(x)\) |

BatchTopK is the simplest nontrivial relaxation of fixed TopK. For a batch of \(n\) samples, it keeps the top \(n\times k\) latent activations across the entire batch, so
\[
\sum_{i=1}^n \|f(x_i)\|_0 = nk,
\]
and the average per-sample \(L_0\) is exactly \(k\), while the individual support sizes \(k_i=\|f(x_i)\|_0\) vary [2412.06410]. The paper explicitly interprets this as allocating more latents to more complex samples and fewer to simpler ones, but there is no separate \(k(x)\) predictor.

“Improving Sparse Autoencoder with Dynamic Attention” replaces TopK entirely by a sparsemax cross-attention encoder. With concept-score vector \(z\), sparsemax computes
\[
p_i=\max(z_i-\tau,0),\qquad \sum_i p_i=1,
\]
so the support size is data-dependent through the threshold \(\tau(z)\) [2604.14925]. The paper explicitly states that sparsemax can be viewed as “a more precise version of BatchTopK, where we set \(K\) at the sample level rather than the batch level,” but its architecture is a cross-attention SAE rather than a standard encoder–TopK–decoder pipeline.

By contrast, some influential TopK-adjacent papers are best read as motivation rather than as AdaptiveK methods. “Unlocking the Address Book” introduces a **Dual-Budget Strategy** for KV-cache interpretation, with \(K_{\text{inference}}=8\) for Keys and shallow layers and \(K_{\text{inference}}=16\) for deep Values, but this is a static, representation-type-dependent policy rather than learned per-token adaptation [2512.10547]. “TopK Language Models” moves fixed TopK sparsity into the transformer itself with \(k=64\), but \(k\) remains a global hyperparameter and does not vary by token, layer, or context [2506.21468].

## 4. Training objectives, regularization, and optimization

AdaptiveK training introduces a second problem beyond ordinary reconstruction: learning or stabilizing the mechanism that determines support size.

The complexity-driven AdaptiveK paper uses a three-phase procedure. First, it trains the ridge probe. Second, it freezes the probe and trains the SAE with
\[
L_{\mathrm{SAE}} = L_{\mathrm{recon}} + \alpha L_{\mathrm{sparsity}} + \beta L_{\mathrm{aux}},
\]
where
\[
L_{\mathrm{recon}}=\|x-\hat{x}\|_2^2,\qquad
L_{\mathrm{sparsity}}=\frac{\|z\|_1}{\|x\|_2}.
\]
Third, it jointly fine-tunes probe and SAE with
\[
L_{\mathrm{joint}} = L_{\mathrm{SAE}} + \gamma(L_{\mathrm{probe}}+\delta L_{\mathrm{deviation}}),
\]
and
\[
L_{\mathrm{deviation}}=|w-w^0|_2+|b-b^0|,
\]
to prevent the probe from drifting too far from the pretrained complexity estimator [2508.17320]. The paper reports \(\alpha=0.005\), \(\beta=1/32\), \(\gamma=0.9\), and an adaptive \(\delta\) between \(0.01\) and \(0.5\).

SoftSAE trains adaptation end to end by replacing discrete cardinality with a differentiable relaxation. Its loss is
\[
\mathcal{L}(x):=\|x-\hat{x}(f_{\mathrm{train}}(x))\|_2^2+\lambda \mathcal{S}(\hat{k})+\gamma \mathcal{L}_{\mathrm{aux}},
\]
where the key regularizer constrains only the **expected** budget,
\[
\mathcal{S}(\hat{k})=\frac{1}{\beta}\ln\!\left(1+e^{(\mathbb{E}[\hat{k}]-k)\beta}\right).
\]
This means \(k\) is not a fixed per-sample support size but a target mean sparsity level, while \(\hat{k}(x)\) is free to vary under a global average constraint [2605.06610]. The auxiliary term is the standard dead-feature prevention loss based on underused neurons reconstructing residual error.

SoftSAE also identifies a training pathology specific to soft adaptive selection. As \(\alpha\to 0\), the soft selector becomes nearly binary, but the model may “hide” information in tiny nonzero weights and thereby bypass the intended sparsity constraint. Its remedy is operationally important: freeze the Dynamic Sparsity MLP near the end of training and replace soft selection with hard TopK [2605.06610].

BatchTopK exposes a different optimization issue: the training rule depends on the other samples in the minibatch. To remove this dependency at inference, the paper estimates a global threshold
\[
\theta = \mathbb{E}_{\mathbf{X}}\left[\min \{z_{i,j}(\mathbf{X}) \mid z_{i,j}(\mathbf{X})>0\}\right]
\]
and replaces BatchTopK by JumpReLU-like thresholding at test time [2412.06410]. The paper is explicit that this is a heuristic approximation rather than an exact match to the training mechanism.

Several later papers suggest complementary, rather than competing, solutions. “Beyond the Hard Budget” adds pre-TopK sparsity regularizers to fixed-\(k\) SAEs, including an off-support \(\ell_1\) penalty and a scale-invariant \(\ell_1/\ell_2\) ratio penalty, both restricted to batch-active units [2606.27321]. SplInterp, from a spline-theoretic perspective, introduces PAM-SGD, an alternating method with exact decoder updates for TopK SAEs and argues that it works well “provided that the sparsity can adapt to the data,” which suggests a natural optimization interface for future AdaptiveK variants [2505.11836].

## 5. Empirical behavior across language and vision

The most direct empirical support for AdaptiveK comes from the complexity-probe results and from the reconstruction frontiers reported by explicit adaptive models.

In the AdaptiveK paper, the linear complexity probe performs comparably to more expressive regressors. The reported scores are RMSE \(1.41\), Pearson \(0.72\), and Spearman \(0.76\) for the linear model, versus RMSE \(1.37\), Pearson \(0.74\), and Spearman \(0.77\) for a one-hidden-layer MLP, and RMSE \(1.42\), Pearson \(0.71\), and Spearman \(0.74\) for XGBoost [2508.17320]. The paper interprets this as evidence that context complexity is largely linearly encoded in LLM activations. It also gives concrete adaptive budgets: predicted complexity \(3.18\to K=95\), \(4.26\to K=137\), \(5.39\to K=187\), \(6.05\to K=216\), \(9.19\to K=298\), and \(8.91\to K=294\) [2508.17320]. Across Pythia-70M, Pythia-160M, and Gemma-2-2B, the paper reports improved reconstruction fidelity, explained variance, and cosine similarity relative to fixed-sparsity baselines, while reducing the need for wide \(K\)-sweeps [2508.17320].

SoftSAE provides the most detailed explicit AdaptiveK benchmark numbers. On CLIP embeddings at target \(k=140\), TopK achieves FVE \(0.933\), BatchTopK \(0.943\), Matryoshka \(0.926\), and SoftSAE \(0.935\); SoftSAE’s realized \(L_0\) is \(149.300\), closer to the target than BatchTopK’s \(152.770\) or Matryoshka’s \(155.405\) [2605.06610]. At \(k=220\), SoftSAE reaches FVE \(0.968\) with \(L_0=223.459\), versus TopK FVE \(0.970\) at exact \(L_0=220\) and BatchTopK FVE \(0.977\) at \(L_0=235.186\) [2605.06610]. On Gemma-2-2B activations at target \(k=320\), SoftSAE reports \(L_0=302.355\), FVE \(0.920\), absorption fraction \(0.029\), SCR top-2 \(0.178\), and TPP top-10 \(0.250\) [2605.06610]. Its strongest claim, however, is qualitative rather than purely reconstructive: \(\hat{k}\) tracks intuitive semantic complexity in images and text.

BatchTopK offers a simpler form of adaptivity and consistently improves over fixed TopK in reconstruction quality at the same average sparsity. The paper states that at \(L_0=32\), BatchTopK has lower NMSE and less CE degradation than TopK on both GPT-2 Small and Gemma 2 2B, and that the per-sample active-count distribution is genuinely broad: some samples activate only a single latent, while some activate more than 80 [2412.06410]. This is strong evidence that adaptive per-sample support size need not collapse back to near-constant \(k\).

The sparsemax attention SAE shows similar qualitative behavior in a different architecture. On OpenWeb with dictionary size \(M=3072\), it reports NMSE \(0.005\) versus TopK \(0.014\) and cross-entropy degradation \(0.031\) versus TopK \(0.209\); on WikiText-103 at the same \(M\), it reports NMSE \(0.008\) versus TopK \(0.024\) and cross-entropy degradation \(-0.106\) versus TopK \(-0.898\) [2604.14925]. In vision, the same paper reports that sparsemax explains class information with far fewer concepts than TopK or BatchTopK in top-\(n\) concept classification, which it interprets as evidence that support size should depend on sample complexity rather than be globally fixed [2604.14925].

## 6. Misconceptions, limitations, and open problems

A persistent misconception is to equate any nonconstant support size with AdaptiveK. The literature is more differentiated than that. BatchTopK is adaptive only because it enforces a batch-level top-\(nk\) budget, not because it learns a dedicated per-example cardinality predictor [2412.06410]. Sparsemax attention infers a variable support set, but does so through simplex projection in a cross-attention SAE rather than through explicit TopK-with-\(k(x)\) gating [2604.14925]. The KV-cache **Dual-Budget Strategy** uses different fixed budgets for Keys and Values, but it is “fixed Top-K with manually chosen asymmetric budgets,” not AdaptiveK in the strict sense [2512.10547].

A second misconception is that explicit AdaptiveK is always the simplest route to better sparse representations. “Beyond the Hard Budget” shows that fixed-\(k\) TopK SAEs can behave more adaptively in effect when pre-TopK activations are regularized, especially with the \(\ell_1/\ell_2\) ratio penalty that concentrates information into fewer effective units and improves robustness to inference-time changes in \(k\) [2606.27321]. Conversely, “AbsTopK” argues that the more fundamental limitation of standard TopK may be nonnegativity rather than fixed support size: by selecting the \(k\) largest-magnitude activations and preserving sign, AbsTopK improves reconstruction and several steering/probing metrics while still using a fixed \(k\) [2510.00404].

Adaptive methods also introduce new engineering costs. SoftSAE notes that differentiable SoftTopK becomes a bottleneck for large dictionaries and reports training times of about 2 hours versus 1.5 for baselines on CLIP and about 6 hours versus 2.5 on Gemma, together with the need to manage temperature annealing, budget annealing, and the soft-to-hard transition [2605.06610]. BatchTopK inherits batch dependence and an inference mismatch due to threshold approximation [2412.06410]. ATM, a time-adaptive masking method designed to reduce feature absorption, achieves an absorption score of \(0.0068\) versus TopK’s \(0.1402\), but does so with reported \(L_0\) sparsity \(3280\) versus TopK’s \(40\), so its stability improvements are not obtained under the same hard sparsity regime [2510.08855].

Finally, there is a theoretical caution. “Toward Identifiable Sparse Autoencoders” does not study AdaptiveK directly, but it argues that even fixed-\(k\) TopK SAEs can be unstable because of one-sided activations, poor dictionary conditioning, and weak amortized encoders [2605.31245]. A plausible implication is that adaptive support size could improve fit while requiring stronger monitoring of support conditioning and code stability, especially if support cardinality itself becomes variable. SplInterp adds a complementary observation: fixed-\(K\) TopK partitions have a clean \(K\)-th order power-diagram geometry, whereas a true AdaptiveK system would likely mix multiple support orders and therefore lose some of that geometric simplicity [2505.11836].

Across these works, AdaptiveK emerges less as a single algorithm than as a design principle: sparse autoencoders should allocate latent support in a way that reflects the heterogeneity of the data. Explicit per-input \(k\)-prediction, differentiable soft top-\(k\), batch-average budgeting, sparsemax support inference, and effective-sparsity regularization are all different realizations of that principle, but they differ sharply in whether they learn **how many** features to use, merely vary the support as a side effect, or retain fixed cardinality while reshaping the code geometry.

Source: https://www.emergentmind.com/topics/adaptive-top-k-sparse-autoencoders-adaptivek