Dynamic Activation N:M Sparsity

• Dynamic Activation N:M Sparsity is a technique that partitions an activation tensor into groups of M and retains the top N elements based on their magnitude to enable efficient sparse GEMM operations.
• The method uses online mask generation, dynamically adapting per input and token, and incorporates mechanisms like error compensation or dense residuals to preserve crucial information.
• Empirical results in LLMs and diffusion models show that this approach can significantly reduce latency and improve throughput while maintaining high accuracy with hardware-aligned sparse patterns.

Searching arXiv for the cited papers to ground the article in current literature. arxiv_search.query({"search_query":"id:(Meng et al., 25 Jun 2026) OR id:(An et al., 4 Aug 2025) OR id:(Yang et al., 2019) OR id:(Madhyastha et al., 5 Feb 2026) OR id:(Ma et al., 2024) OR id:(Awasthi et al., 2024)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}) arxiv_search.search({"query":"(Meng et al., 25 Jun 2026)","max_results":5}) Dynamic activation N:M sparsity is a form of semi-structured sparsity in which an activation tensor is partitioned into contiguous groups of size $M$, and only $N$ elements in each group are retained at run time. In the literature surveyed here, the defining distinction is that the mask is produced online from the current activations, so the active subset varies across inputs, tokens, layers, or diffusion steps, unlike static N:M weight sparsity, whose mask is fixed offline (Meng et al., 25 Jun 2026). In this setting, “activation sparsity” often means that large values are sparse in magnitude rather than that activations contain many exact zeros, and the central technical problem is to convert that input-dependent structure into hardware-aligned sparse GEMM without losing the information carried by moderate values (Meng et al., 25 Jun 2026).

1. Formal definition and scope

For an activation tensor $\mathbf{X}$, dynamic activation N:M sparsity introduces an input-dependent mask $\mathbf{M}(\mathbf{X})$ drawn from a constraint set $\mathcal{S}_{N\!:\!M}$ such that, in every hardware block $b$ of size $M$, $\|\mathbf{M}_b\|_0 = N$, and the sparsified activation is

$$\mathbf{X}_{\mathrm{sp}} = \mathbf{M}(\mathbf{X}) \odot \mathbf{X}, \qquad \mathbf{M}(\mathbf{X}) \in \mathcal{S}_{N\!:\!M}.$$

This formalization appears explicitly in recent LLM inference work, where the mask is recomputed for every activation tensor, token, and batch, and must obey accelerator-supported patterns such as 4:8 in pairs for NVFP4 and 2:4 for HiF4 (Meng et al., 25 Jun 2026).

The same basic definition is used in training-free prefill sparsification for LLMs: for an activation tensor $X \in \mathbb{R}^{B \times d_{\text{in}}}$, the hidden dimension is partitioned into contiguous groups of size $N$0, and in each group only $N$1 activations are retained (An et al., 4 Aug 2025). Earlier work on dynamic activation sparsity framed the idea less as fixed hardware blocks and more as run-time winner selection, but the underlying principle is the same: each input induces a new activation mask, while the layer architecture remains unchanged (Yang et al., 2019).

Dynamic activation N:M sparsity therefore sits at the intersection of three constraints. It is input-adaptive, because masks depend on current activations; semi-structured, because each local group must satisfy an exact $N$2-of-$N$3 rule; and systems-constrained, because the grouping dimension, block size, and metadata layout must match sparse GEMM kernels. A recurrent theme across the literature is that the method is useful only when these three constraints are addressed simultaneously (Meng et al., 25 Jun 2026).

2. Online mask generation and selection rules

The simplest dynamic rule is local top-$N$4 selection by magnitude. In SharQ, each hardware block is processed independently: for NVFP4 4:8 in-pairs, each 8-element region is organized as 4 pairs and the method selects the 4 elements corresponding to the 2 pairs with the largest magnitude under a pair-topk rule; for HiF4 2:4, it selects the 2 largest-magnitude elements in each 4-element block (Meng et al., 25 Jun 2026). This is an explicitly online procedure, because the mask depends on the current activation tensor.

Amber Pruner uses a different scoring rule for the same per-group top-$N$5 decision. Instead of ranking activations only by $N$6, it defines a weight-aware score

$N$7

where the channel factor $N$8 is computed once offline from robustly normalized weights. At run time, the actual N:M mask remains dynamic because $N$9 changes with the prompt, batch, and token, so the top-$\mathbf{X}$0 positions within each $\mathbf{X}$1-group change on every forward pass (An et al., 4 Aug 2025). This formulation makes the mask partly static in its coefficients and fully dynamic in its realized pattern.

Earlier dynamic activation work used winner-take-all rather than hardware-grouped top-$\mathbf{X}$2. DASNet computes original activations $\mathbf{X}$3, ranks neurons or channels by activation strength, retains a subset of “winner” units, and masks the rest. In fully connected layers the winner set is chosen from activation magnitudes, and in convolutional layers the selection is channel-wise, with feature vectors defined by per-channel mean or maximum activation and a winner rate

$\mathbf{X}$4

This is structured activation sparsity, but its grouping is coarse, typically whole-neuron or whole-channel rather than small fixed hardware blocks (Yang et al., 2019).

A more elaborate dynamic construction appears in FFN training for LLMs. Venom applies neuron-level routing to activations after SquaredReLU: activations are partitioned into $\mathbf{X}$5 submatrices, 4 columns are retained, and then 2:4 pruning is applied within those active columns. The active columns are selected dynamically by routing tokens to experts based on their current activations, so the effective activation mask changes every forward pass (Madhyastha et al., 5 Feb 2026). This places dynamic activation N:M sparsity closer to MoE-style routing than to purely local thresholding.

3. Sparsity as routing, compensation, and quantization-aware decomposition

A central recent development is the treatment of activation sparsity not as irreversible dropping but as a routing mechanism coupled to error compensation. SharQ makes this explicit: “activation sparsification is routing, not dropping” (Meng et al., 25 Jun 2026). Given $\mathbf{X}$6, it forms a sparse backbone

$\mathbf{X}$7

quantizes that backbone for the sparse path,

$\mathbf{X}$8

and then defines the dense residual relative to the quantized sparse backbone,

$\mathbf{X}$9

This residual contains both sparsification loss and sparse-path quantization loss, so the final approximation reduces to reconstructing $\mathbf{M}(\mathbf{X})$0 well in a dense FP4 path (Meng et al., 25 Jun 2026).

The significance of this construction is distributional. Block-scaled FP4 formats choose scales from local maxima, so a few outliers can dominate the scale and degrade the resolution available to moderate values. SharQ’s dynamic N:M mask routes outliers into a sparse backbone and leaves a denser residual whose local dynamic range is more compact, allowing the sparse and dense FP4 paths to use different scale views matched to different statistics (Meng et al., 25 Jun 2026). A plausible summary is that the method decouples “which values should be routed to a sparse accelerator path” from “which values are still needed numerically after quantization.”

Amber Pruner addresses the same interaction from the W8A8 side. In Outstanding-sparse, SmoothQuant scaling is inverted,

$\mathbf{M}(\mathbf{X})$1

so that the activation range is expanded rather than compressed, exposing more structure for structured top-$\mathbf{M}(\mathbf{X})$2 selection while weights remain quantized (An et al., 4 Aug 2025). Operationally, activations are scaled, quantized, scored using weight-aware coefficients, pruned by N:M top-$\mathbf{M}(\mathbf{X})$3, and then consumed by sparse-dense GEMM. Here again, dynamic activation sparsity is not used in isolation; it is embedded in a quantization pipeline whose distributional assumptions must be adjusted for sparsity.

The same design logic appears in diffusion transformers. RT-Lynx first applies 2:4 structured top-$\mathbf{M}(\mathbf{X})$4 to activations, then rescales the retained values by a norm-compensation factor

$\mathbf{M}(\mathbf{X})$5

and finally adds a LoRA residual branch,

$\mathbf{M}(\mathbf{X})$6

The sparse branch captures the dominant activation structure; the low-rank residual branch recovers information lost by structured pruning (Cong et al., 26 May 2026).

4. Representative realizations across model families

Dynamic activation N:M sparsity has been instantiated in substantially different regimes: early structured dropout, training-free prefill pruning, FP4 inference decomposition, FFN routing during pretraining, and diffusion-model linear-layer acceleration. The common thread is online mask generation; the main differences are granularity, auxiliary compensation, and whether the method targets inference or training.

System	Primary setting	Dynamic N:M mechanism
DASNet (Yang et al., 2019)	CNN and FC inference/training	Winner-take-all neuron or channel masking
Amber Pruner (An et al., 4 Aug 2025)	LLM prefill	Per-token, per-layer top-$\mathbf{M}(\mathbf{X})$7 activation pruning
SharQ (Meng et al., 25 Jun 2026)	LLM FP4 inference	Input-adaptive N:M sparse backbone plus dense residual
Venom (Madhyastha et al., 5 Feb 2026)	LLM FFN pretraining	Neuron-level routing plus 2:4 within active columns
RT-Lynx (Cong et al., 26 May 2026)	Diffusion transformers	Online 2:4 activation Top-K with norm compensation

In LLM serving, Amber Pruner targets the prefill stage, where linear projection layers dominate compute. It applies dynamic N:M activation sparsity to $\mathbf{M}(\mathbf{X})$8, but combines this with layer skipping, because some projections are markedly more sensitive than others (An et al., 4 Aug 2025). SharQ instead replaces dense FP16 or FP8 linear layers with a two-path sparse+dense FP4 decomposition that relies on an online N:M mask and a fused preparation kernel (Meng et al., 25 Jun 2026).

In diffusion transformers, RT-Lynx advocates a “paradigm shift from weight sparsification to activation sparsification,” motivated by the finding that DiT activations are intrinsically sparse and much more robust to N:M sparsification than weights (Cong et al., 26 May 2026). Its sparse masks are recomputed per token, per layer, and per diffusion step, so the N:M pattern is dynamic both across prompts and across denoising time.

At a more abstract level, D2DMoE shows that dynamic activation sparsity can also be understood as dynamic expert selection. It converts dense FFNs to dynamic-$\mathbf{M}(\mathbf{X})$9 MoE layers and uses a threshold rule

$\mathcal{S}_{N\!:\!M}$0

so the number of executed experts varies per token (Szatkowski et al., 2023). This is not explicit N:M block sparsity, but it is a closely related expert-level form of dynamic activation sparsity in which the effective $\mathcal{S}_{N\!:\!M}$1 changes with the input.

5. Empirical behavior, speed–accuracy trade-offs, and common pitfalls

The most favorable recent results come from systems that pair dynamic activation N:M sparsity with explicit error compensation. SharQ, evaluated on Llama‑3.1‑8B, Qwen2.5‑7B, Qwen3‑30B‑A3B, and Qwen3‑VL‑8B, recovers 43–63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, delivers 2.2–2.4$\mathcal{S}_{N\!:\!M}$2 latency reduction over FP16 and 1.2–1.4$\mathcal{S}_{N\!:\!M}$3 throughput improvement over FP8 in LLM serving on an RTX 5090, and reaches up to 1.58$\mathcal{S}_{N\!:\!M}$4 speedup on Wan2.2‑T2V‑A14B video generation when combined with SageAttention (Meng et al., 25 Jun 2026). Amber Pruner reports that it can sparsify and accelerate more than 55% of linear computations during prefill, with average zero-shot accuracy drops at 8:16 of about 0.7% for LLaMA3.1‑8B, 0.8% for Qwen2‑7B, and 0.4% for Qwen3‑30B‑A3B, while naïve top-k activation sparsity causes much larger losses (An et al., 4 Aug 2025).

Diffusion transformers show the same asymmetry between activation and weight sparsity. RT-Lynx reports that naive 2:4 weight sparsity in DiTs can severely degrade FID and ImageReward, whereas activation sparsity is much more robust; with norm compensation and LoRA, the method preserves generation quality while achieving up to a 1.55$\mathcal{S}_{N\!:\!M}$5 speedup on average in linear layers (Cong et al., 26 May 2026). The empirical lesson is that N:M structure is usually tolerable when it follows intrinsic activation sparsity, and much less so when it is imposed directly on dense, information-bearing weights.

The main misconception corrected by the literature is that dynamic activation sparsity automatically translates to practical speedup. In LLaMA models, threshold-based activation truncation can produce substantial nominal sparsity, but predictor-based dynamic schemes underperform sharply when sparsity is pushed aggressively. With a model-wise universal threshold at CETT $\mathcal{S}_{N\!:\!M}$6, average MLP sparsity reaches about 57.13% for LLaMA‑2‑7B and 67.29% for LLaMA‑3‑8B, yet linear predictors recover only about 29.01% and 13.08% predicted sparsity, and topK forcing of higher sparsity sharply reduces recall and downstream task performance (Ma et al., 2024). The same study shows that KV-cache skipping is especially fragile: skipping both $\mathcal{S}_{N\!:\!M}$7 and $\mathcal{S}_{N\!:\!M}$8 for masked heads nearly collapses GSM8k generation accuracy, and even skipping $\mathcal{S}_{N\!:\!M}$9 alone is highly destructive (Ma et al., 2024). Dynamic activation sparsity is therefore not merely a masking problem; it is a joint problem of prediction quality, architectural sensitivity, and memory semantics.

A second misconception is that all tensors can be pruned with the same criterion. For neural gradients, minimum-MSE N:M pruning “catastrophically fails,” whereas unbiased minimum-variance pruning masks preserve training performance; in most cases, 1:2 gradient sparsity is sufficient for training, and 2:4 is usually enough when it is not (Chmiel et al., 2022). This result does not directly concern activations, but it sharply delineates the boundary between activation-side approximation, where magnitude criteria are often effective, and gradient-side approximation, where unbiasedness becomes primary.

6. Relation to adjacent paradigms and open questions

Dynamic activation N:M sparsity differs from static weight sparsity in both optimization and systems assumptions. Static N:M weight pruning learns or selects one mask offline and reuses it for all inputs, whereas dynamic activation N:M sparsity is per-activation, per-token, per-layer, or per-step (Meng et al., 25 Jun 2026). It also differs from purely unstructured activation sparsity, because the latter does not guarantee a fixed local pattern and therefore does not map as directly to sparse tensor-core kernels (An et al., 4 Aug 2025). At the same time, the topic overlaps with MoE routing, dynamic-$b$0 expert execution, WTA dropout, and neuron-level activation functions such as Venom, all of which can be read as structured forms of conditional activation (Szatkowski et al., 2023).

Theoretical work gives this empirical program a formal backdrop. A class of sparsely activated one-hidden-layer networks,

$b$1

is defined by the property that for every input, at most $b$2 hidden units are active. Under the uniform distribution, such classes admit low-degree learning guarantees and improved sample-complexity behavior relative to dense counterparts, while still remaining expressive enough to represent hard functions under arbitrary distributions (Awasthi et al., 2024). This suggests that dynamic activation sparsity is not merely an engineering heuristic; it can also alter the effective complexity of the learned function class.

The open questions in the surveyed papers are mainly systems and co-design questions. SharQ identifies hardware dependence, the overhead of the additional dense residual GEMM, and the role of mask granularity and block size as immediate limitations (Meng et al., 25 Jun 2026). Amber Pruner highlights the lack of SpMM support for activation-side N:M patterns on current general-purpose hardware (An et al., 4 Aug 2025). RT-Lynx shows that fused, online sparse kernels are essential if dynamic Top‑K overhead is not to dominate the sparse path (Cong et al., 26 May 2026). A plausible implication is that future progress will depend less on discovering that activations are sparse, which is already well established, and more on designing kernels, memory formats, and training or distillation procedures that make that sparsity numerically safe and operationally cheap. Transposable N:M mask generation, presently developed for weights, points to one possible direction for activation-side extensions when backward compatibility becomes a first-class requirement (Meng et al., 29 May 2025).