---
title: Dynamic Activation N:M Sparsity
url: https://www.emergentmind.com/topics/dynamic-activation-n-m-sparsity
type: topic
---

# Dynamic Activation N:M Sparsity

Searching arXiv for the cited papers to ground the article in current literature.
arxiv_search.query({"search_query":"id:2606.26587 OR id:2508.02128 OR id:1909.06964 OR id:2602.06183 OR id:2405.09274 OR id:2406.17989","max_results":10,"sort_by":"submittedDate","sort_order":"descending"})
arxiv_search.search({"query":"2606.26587","max_results":5})
Dynamic activation N:M sparsity is a form of semi-structured sparsity in which an activation tensor is partitioned into contiguous groups of size \(M\), and only \(N\) elements in each group are retained at run time. In the literature surveyed here, the defining distinction is that the mask is produced online from the current activations, so the active subset varies across inputs, tokens, layers, or diffusion steps, unlike static N:M weight sparsity, whose mask is fixed offline [2606.26587]. In this setting, “activation sparsity” often means that large values are sparse in magnitude rather than that activations contain many exact zeros, and the central technical problem is to convert that input-dependent structure into hardware-aligned sparse GEMM without losing the information carried by moderate values [2606.26587].

## 1. Formal definition and scope

For an activation tensor \(\mathbf{X}\), dynamic activation N:M sparsity introduces an input-dependent mask \(\mathbf{M}(\mathbf{X})\) drawn from a constraint set \(\mathcal{S}_{N\!:\!M}\) such that, in every hardware block \(b\) of size \(M\), \(\|\mathbf{M}_b\|_0 = N\), and the sparsified activation is
\[
\mathbf{X}_{\mathrm{sp}} = \mathbf{M}(\mathbf{X}) \odot \mathbf{X}, \qquad \mathbf{M}(\mathbf{X}) \in \mathcal{S}_{N\!:\!M}.
\]
This formalization appears explicitly in recent LLM inference work, where the mask is recomputed for every activation tensor, token, and batch, and must obey accelerator-supported patterns such as 4:8 in pairs for NVFP4 and 2:4 for HiF4 [2606.26587].

The same basic definition is used in training-free prefill sparsification for LLMs: for an activation tensor \(X \in \mathbb{R}^{B \times d_{\text{in}}}\), the hidden dimension is partitioned into contiguous groups of size \(M\), and in each group only \(N\) activations are retained [2508.02128]. Earlier work on dynamic activation sparsity framed the idea less as fixed hardware blocks and more as run-time winner selection, but the underlying principle is the same: each input induces a new activation mask, while the layer architecture remains unchanged [1909.06964].

Dynamic activation N:M sparsity therefore sits at the intersection of three constraints. It is **input-adaptive**, because masks depend on current activations; **semi-structured**, because each local group must satisfy an exact \(N\)-of-\(M\) rule; and **systems-constrained**, because the grouping dimension, block size, and metadata layout must match sparse GEMM kernels. A recurrent theme across the literature is that the method is useful only when these three constraints are addressed simultaneously [2606.26587].

## 2. Online mask generation and selection rules

The simplest dynamic rule is local top-\(k\) selection by magnitude. In SharQ, each hardware block is processed independently: for NVFP4 4:8 in-pairs, each 8-element region is organized as 4 pairs and the method selects the 4 elements corresponding to the 2 pairs with the largest magnitude under a pair-topk rule; for HiF4 2:4, it selects the 2 largest-magnitude elements in each 4-element block [2606.26587]. This is an explicitly online procedure, because the mask depends on the current activation tensor.

Amber Pruner uses a different scoring rule for the same per-group top-\(N\) decision. Instead of ranking activations only by \(|X_{ij}|\), it defines a weight-aware score
\[
S_{ij}^* = |X_{ij}| \cdot f(\hat{W}_{:,j}),
\]
where the channel factor \(f(\hat{W}_{:,j})\) is computed once offline from robustly normalized weights. At run time, the actual N:M mask remains dynamic because \(X\) changes with the prompt, batch, and token, so the top-\(N\) positions within each \(M\)-group change on every forward pass [2508.02128]. This formulation makes the mask partly static in its coefficients and fully dynamic in its realized pattern.

Earlier dynamic activation work used winner-take-all rather than hardware-grouped top-\(N\). DASNet computes original activations \(A_{o,l}\), ranks neurons or channels by activation strength, retains a subset of “winner” units, and masks the rest. In fully connected layers the winner set is chosen from activation magnitudes, and in convolutional layers the selection is channel-wise, with feature vectors defined by per-channel mean or maximum activation and a winner rate
\[
p = \frac{N_{\text{winner}}}{N_{\text{total}}}.
\]
This is structured activation sparsity, but its grouping is coarse, typically whole-neuron or whole-channel rather than small fixed hardware blocks [1909.06964].

A more elaborate dynamic construction appears in FFN training for LLMs. Venom applies neuron-level routing to activations after SquaredReLU: activations are partitioned into \(\left[\frac{n_{\text{rows}}}{V}, M\right]\) submatrices, 4 columns are retained, and then 2:4 pruning is applied within those active columns. The active columns are selected dynamically by routing tokens to experts based on their current activations, so the effective activation mask changes every forward pass [2602.06183]. This places dynamic activation N:M sparsity closer to MoE-style routing than to purely local thresholding.

## 3. Sparsity as routing, compensation, and quantization-aware decomposition

A central recent development is the treatment of activation sparsity not as irreversible dropping but as a routing mechanism coupled to error compensation. SharQ makes this explicit: “activation sparsification is routing, not dropping” [2606.26587]. Given \(\mathbf{X}\), it forms a sparse backbone
\[
\mathbf{X}_{\mathrm{sp}} = \mathbf{M}(\mathbf{X}) \odot \mathbf{X},
\]
quantizes that backbone for the sparse path,
\[
\widetilde{\mathbf{X}}_{\mathrm{sp}} = \mathcal{D}_{\mathrm{sp}}\!\left(\mathcal{Q}_{\mathrm{sp}}(\mathbf{X}_{\mathrm{sp}})\right),
\]
and then defines the dense residual relative to the **quantized** sparse backbone,
\[
\mathbf{R} = \mathbf{X} - \widetilde{\mathbf{X}}_{\mathrm{sp}}.
\]
This residual contains both sparsification loss and sparse-path quantization loss, so the final approximation reduces to reconstructing \(\mathbf{R}\) well in a dense FP4 path [2606.26587].

The significance of this construction is distributional. Block-scaled FP4 formats choose scales from local maxima, so a few outliers can dominate the scale and degrade the resolution available to moderate values. SharQ’s dynamic N:M mask routes outliers into a sparse backbone and leaves a denser residual whose local dynamic range is more compact, allowing the sparse and dense FP4 paths to use different scale views matched to different statistics [2606.26587]. A plausible summary is that the method decouples “which values should be routed to a sparse accelerator path” from “which values are still needed numerically after quantization.”

Amber Pruner addresses the same interaction from the W8A8 side. In Outstanding-sparse, SmoothQuant scaling is inverted,
\[
\hat{s}_j = \frac{1}{s_j},
\]
so that the activation range is expanded rather than compressed, exposing more structure for structured top-\(N\) selection while weights remain quantized [2508.02128]. Operationally, activations are scaled, quantized, scored using weight-aware coefficients, pruned by N:M top-\(N\), and then consumed by sparse-dense GEMM. Here again, dynamic activation sparsity is not used in isolation; it is embedded in a quantization pipeline whose distributional assumptions must be adjusted for sparsity.

The same design logic appears in diffusion transformers. RT-Lynx first applies 2:4 structured top-\(K\) to activations, then rescales the retained values by a norm-compensation factor
\[
s = \sqrt{\frac{\|\mathbf{X}\|_2^2}{\|\tilde{\mathbf{X}}\|_2^2 + \epsilon}},
\qquad
S(\mathbf{X}) = s \cdot \tilde{\mathbf{X}},
\]
and finally adds a LoRA residual branch,
\[
\mathbf{Y} = S(\mathbf{X}) \cdot \mathbf{W}^{\top} + \mathbf{X} \cdot (\mathbf{L}_A\mathbf{L}_B)^{\top}.
\]
The sparse branch captures the dominant activation structure; the low-rank residual branch recovers information lost by structured pruning [2605.26632].

## 4. Representative realizations across model families

Dynamic activation N:M sparsity has been instantiated in substantially different regimes: early structured dropout, training-free prefill pruning, FP4 inference decomposition, FFN routing during pretraining, and diffusion-model linear-layer acceleration. The common thread is online mask generation; the main differences are granularity, auxiliary compensation, and whether the method targets inference or training.

| System | Primary setting | Dynamic N:M mechanism |
|---|---|---|
| DASNet [1909.06964] | CNN and FC inference/training | Winner-take-all neuron or channel masking |
| Amber Pruner [2508.02128] | LLM prefill | Per-token, per-layer top-\(N\) activation pruning |
| SharQ [2606.26587] | LLM FP4 inference | Input-adaptive N:M sparse backbone plus dense residual |
| Venom [2602.06183] | LLM FFN pretraining | Neuron-level routing plus 2:4 within active columns |
| RT-Lynx [2605.26632] | Diffusion transformers | Online 2:4 activation Top-K with norm compensation |

In LLM serving, Amber Pruner targets the prefill stage, where linear projection layers dominate compute. It applies dynamic N:M activation sparsity to \(q_{\text{proj}}, k_{\text{proj}}, v_{\text{proj}}, o_{\text{proj}}, gate_{\text{proj}}, up_{\text{proj}}, down_{\text{proj}}\), but combines this with layer skipping, because some projections are markedly more sensitive than others [2508.02128]. SharQ instead replaces dense FP16 or FP8 linear layers with a two-path sparse+dense FP4 decomposition that relies on an online N:M mask and a fused preparation kernel [2606.26587].

In diffusion transformers, RT-Lynx advocates a “paradigm shift from weight sparsification to activation sparsification,” motivated by the finding that DiT activations are intrinsically sparse and much more robust to N:M sparsification than weights [2605.26632]. Its sparse masks are recomputed per token, per layer, and per diffusion step, so the N:M pattern is dynamic both across prompts and across denoising time.

At a more abstract level, D2DMoE shows that dynamic activation sparsity can also be understood as dynamic expert selection. It converts dense FFNs to dynamic-\(k\) MoE layers and uses a threshold rule
\[
G(z)_i =
\begin{cases}
1 & \text{if } R(z)_i \ge \tau \cdot \max_j R(z)_j,\\
0 & \text{otherwise},
\end{cases}
\]
so the number of executed experts varies per token [2310.04361]. This is not explicit N:M block sparsity, but it is a closely related expert-level form of dynamic activation sparsity in which the effective \(N\) changes with the input.

## 5. Empirical behavior, speed–accuracy trade-offs, and common pitfalls

The most favorable recent results come from systems that pair dynamic activation N:M sparsity with explicit error compensation. SharQ, evaluated on Llama‑3.1‑8B, Qwen2.5‑7B, Qwen3‑30B‑A3B, and Qwen3‑VL‑8B, recovers 43–63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, delivers 2.2–2.4\(\times\) latency reduction over FP16 and 1.2–1.4\(\times\) throughput improvement over FP8 in language model serving on an RTX 5090, and reaches up to 1.58\(\times\) speedup on Wan2.2‑T2V‑A14B video generation when combined with SageAttention [2606.26587]. Amber Pruner reports that it can sparsify and accelerate more than 55% of linear computations during prefill, with average zero-shot accuracy drops at 8:16 of about 0.7% for LLaMA3.1‑8B, 0.8% for Qwen2‑7B, and 0.4% for Qwen3‑30B‑A3B, while naïve top-k activation sparsity causes much larger losses [2508.02128].

Diffusion transformers show the same asymmetry between activation and weight sparsity. RT-Lynx reports that naive 2:4 weight sparsity in DiTs can severely degrade FID and ImageReward, whereas activation sparsity is much more robust; with norm compensation and LoRA, the method preserves generation quality while achieving up to a 1.55\(\times\) speedup on average in linear layers [2605.26632]. The empirical lesson is that N:M structure is usually tolerable when it follows intrinsic activation sparsity, and much less so when it is imposed directly on dense, information-bearing weights.

The main misconception corrected by the literature is that dynamic activation sparsity automatically translates to practical speedup. In LLaMA models, threshold-based activation truncation can produce substantial nominal sparsity, but predictor-based dynamic schemes underperform sharply when sparsity is pushed aggressively. With a model-wise universal threshold at CETT \(=0.2\), average MLP sparsity reaches about 57.13% for LLaMA‑2‑7B and 67.29% for LLaMA‑3‑8B, yet linear predictors recover only about 29.01% and 13.08% predicted sparsity, and topK forcing of higher sparsity sharply reduces recall and downstream task performance [2405.09274]. The same study shows that KV-cache skipping is especially fragile: skipping both \(K\) and \(V\) for masked heads nearly collapses GSM8k generation accuracy, and even skipping \(K\) alone is highly destructive [2405.09274]. Dynamic activation sparsity is therefore not merely a masking problem; it is a joint problem of prediction quality, architectural sensitivity, and memory semantics.

A second misconception is that all tensors can be pruned with the same criterion. For neural gradients, minimum-MSE N:M pruning “catastrophically fails,” whereas unbiased minimum-variance pruning masks preserve training performance; in most cases, 1:2 gradient sparsity is sufficient for training, and 2:4 is usually enough when it is not [2203.10991]. This result does not directly concern activations, but it sharply delineates the boundary between activation-side approximation, where magnitude criteria are often effective, and gradient-side approximation, where unbiasedness becomes primary.

## 6. Relation to adjacent paradigms and open questions

Dynamic activation N:M sparsity differs from static weight sparsity in both optimization and systems assumptions. Static N:M weight pruning learns or selects one mask offline and reuses it for all inputs, whereas dynamic activation N:M sparsity is per-activation, per-token, per-layer, or per-step [2606.26587]. It also differs from purely unstructured activation sparsity, because the latter does not guarantee a fixed local pattern and therefore does not map as directly to sparse tensor-core kernels [2508.02128]. At the same time, the topic overlaps with MoE routing, dynamic-\(k\) expert execution, WTA dropout, and neuron-level activation functions such as Venom, all of which can be read as structured forms of conditional activation [2310.04361].

Theoretical work gives this empirical program a formal backdrop. A class of sparsely activated one-hidden-layer networks,
\[
\mathcal{H}_{n,s,k},
\]
is defined by the property that for every input, at most \(k\) hidden units are active. Under the uniform distribution, such classes admit low-degree learning guarantees and improved sample-complexity behavior relative to dense counterparts, while still remaining expressive enough to represent hard functions under arbitrary distributions [2406.17989]. This suggests that dynamic activation sparsity is not merely an engineering heuristic; it can also alter the effective complexity of the learned function class.

The open questions in the surveyed papers are mainly systems and co-design questions. SharQ identifies hardware dependence, the overhead of the additional dense residual GEMM, and the role of mask granularity and block size as immediate limitations [2606.26587]. Amber Pruner highlights the lack of SpMM support for activation-side N:M patterns on current general-purpose hardware [2508.02128]. RT-Lynx shows that fused, online sparse kernels are essential if dynamic Top‑K overhead is not to dominate the sparse path [2605.26632]. A plausible implication is that future progress will depend less on discovering that activations are sparse, which is already well established, and more on designing kernels, memory formats, and training or distillation procedures that make that sparsity numerically safe and operationally cheap. Transposable N:M mask generation, presently developed for weights, points to one possible direction for activation-side extensions when backward compatibility becomes a first-class requirement [2505.23949].

Source: https://www.emergentmind.com/topics/dynamic-activation-n-m-sparsity