--- title: Semi-Structured N:M Sparsity url: https://www.emergentmind.com/topics/semi-structured-n-m-sparsity type: topic --- # Semi-Structured N:M Sparsity Semi-structured N:M sparsity is a fine-grained structural pruning paradigm for neural network weights or activations, in which every contiguous group (block) of M elements is constrained to have at most or exactly N nonzero entries. This approach enables a balance between the flexibility and accuracy retention of unstructured pruning and the hardware efficiency of coarse-grained structured pruning. Modern accelerators, especially NVIDIA's Sparse Tensor Cores, natively support common N:M patterns such as 2:4, allowing for significant computational speedups while maintaining high model fidelity. N:M sparsity is applicable to both weights and activations and is central to state-of-the-art compression and acceleration techniques for deep neural networks, especially large language models and computer vision models [2209.07617, 2312.07061, 2503.01253]. ## 1. Mathematical Foundation and Constraint Formalism Let $W\in\mathbb{R}^{d\times m}$ be a weight matrix of a fully connected or convolutional layer. The N:M semi-structured sparsity constraint partitions each row into $B = \lceil m/M \rceil$ disjoint, contiguous blocks of size $M$ [2209.07617, 2407.20496]. Within every block $b$, exactly $N$ of the $M$ weights are nonzero, enforced via a binary mask $M\in\{0,1\}^{d\times m}$: \[ \forall i \in \{1,\ldots,d\},\;\forall b\in\{1,\ldots,B\}:~ \|M_{i,b}\|_0 = N \] This structure is enforced for weights, and similarly, semi-structured sparsity can be defined for activations by applying the same blockwise $N$-nonzero constraint to each block in the activation vector [2509.22166, 2508.02128]. The semi-structured pattern allows arbitrary choice of the $N$ nonzeros within each $M$-sized block, distinguishing it from global unstructured sparsity (maximum flexibility, poor hardware mapping) and from block/channel/pruning (low flexibility, high hardware efficiency). ## 2. Practical Algorithms for N:M Sparsification Contemporary methods for enforcing N:M sparsity fall into "rule-based," "combinatorial/continuous optimization," and "probabilistic" frameworks: - **Blockwise Top-N Magnitude Pruning:** The simplest strategy selects the $N$ largest-magnitude weights per $M$-block, updating the mask after each pruning cycle [2209.07617, 2509.08747]. - **Mask Decay and Structure Decay (Decaying Pruning):** Schedules such as "pruning mask decay" introduce a decay parameter $\alpha(t)$ on pruned weights, allowing gradients to flow and weights to vanish smoothly; "structure decay" (decay schedule $\beta(t)$) softens blockwise importance scores over time, enabling more exploration of block configurations in early training [2209.07617]. - **ADMM-based Constrained Optimization:** The NxMTransformer framework formulates the N:M constraint as a joint optimization using Alternating Direction Method of Multipliers (ADMM). At each iteration, a step projects the weight tensor onto the feasible mask set (per-block top-N selection) [2110.15766]. - **Combinatorial and Probabilistic Mask Learning:** Divide-and-conquer approaches (LBC) enumerate all $\binom{M}{N}$ possible kept-weight combinations per block, learning scores over candidates and gradually pruning away low-scoring subsets with scheduled annealing and a straight-through estimator (STE) [2206.06662]. In large models, MaskLLM and MaskPro use learnable distributions over the mask space via Gumbel-Softmax [2409.17481] or categorical sampling with REINFORCE and variance reduction [2506.12876] to enable end-to-end differentiability and practical scaling. - **Dependency-Aware Pruning:** In multi-projection MLPs (e.g., LLMs with SwiGLU), importance scores are calibrated by both weight magnitude and the activation norm of corresponding dependency groups, boosting signal for weights coupled to high-energy neurons [2405.01943]. - **Multi-Axis and Hierarchical Selection:** MaxQ performs sequential multi-axis block selection (e.g., filter and kernel axes in CNNs) with soft mask generation and gradual sparsity ramp-up for improved block selection and accuracy [2312.07061]. Hierarchical N:M (HiNM) pruning further composes coarse-grained vector sparsity with fine-grained rowwise N:M, necessitating specialized output/input channel permutation (gyro-permutation) for accuracy recovery [2407.20496]. ## 3. Activation Sparsity and Training-Free Techniques Recent work has extended N:M sparsity to activation tensors for runtime compression and dynamic, input-adaptive sparsity. Training-free algorithms such as Amber Pruner apply calibrated channel norms—derived from weight statistics—to weight the activation magnitudes before top-N blockwise pruning, optionally skipping highly sensitive layers by precomputed perturbation error [2508.02128]. Lightweight error mitigation strategies include per-block threshold tuning, layerwise scale correction, and residual Taylor expansion (to correct for pruned activation contributions in downstream layers) [2509.22166]. The selection of block size and the per-block $N$ (e.g., 2:4, 8:16) is tuned for hardware alignment, with 8:16 achieving a favorable accuracy/speed trade-off on 16-lane vector units. Practical guidelines suggest using small held-out calibration sets for tuning and selecting sparsity not exceeding the hardware's theoretical acceleration by more than ~20%. ## 4. Hardware Mapping and Algorithmic Acceleration N:M sparsity is particularly well-suited for recent hardware: - **NVIDIA Sparse Tensor Cores** support 2:4 sparsity at the hardware level, accepting block masks and natively skipping zero computations, yielding speedups approaching $M/N$ in compute-bound settings [2209.07617, 2503.01253]. Support for additional block sizes (4:8, 8:16) is emerging [2407.20496, 2508.02128]. - **Data Layout and Compression:** Compressed storage involves retaining N values and $\log_2\binom{M}{N}$ bits of metadata per block. Hardware mapping requires prepacking weights and sometimes activations, and per-layer memory access is optimized by hierarchical blocking and decompression only as needed [2503.01253]. - **Training/Inference Acceleration:** Structured kernels exploit regular patterns for coalesced memory access and vectorized execution. NM-SpMM, for example, implements hierarchical blocking and pipelined memory operations to achieve up to $6.3\times$ speedup over cuBLAS dense GEMM and $2.1\times$ over prior N:M-sparse libraries at high sparsity ratios (e.g., 87.5%, 1:8) [2503.01253]. - **Transposable N:M Masks:** For accelerating both the forward and backward pass (where $W^\top$ is used), transposable N:M sparsity imposes additional row/column cardinality constraints in each M×M block. The TSENOR framework solves mask generation as an optimal transport problem with tensorized Dykstra projections and efficient rounding for scalability [2505.23949]. ## 5. Trade-Offs: Accuracy, Compression, and Security N:M sparsity offers a favorable accuracy/performance trade-off at moderate sparsity. For example, Transformer Big trained with 2:4 sparsity achieves <0.1 BLEU loss versus dense at 2× FLOP reduction [2209.07617], ResNet-50 with 1:4 or 2:4 sparsity matches or outperforms dense top-1 accuracy, and LLaMA2-7B under 2:4 with CAST achieves only +0.09 perplexity at a 0.36% gain in zero-shot accuracy compared to dense model [2509.25996]. Higher sparsity ratios and larger block sizes (e.g., 1:16, 16:32) are possible but risk more substantial accuracy drops unless mitigated with advanced mask learning and calibration [2206.06662, 2505.23949]. Moreover, combining N:M sparsity with low-bit quantization (e.g., 1.58-bit BitNet) is synergistic: ternarized models exhibit higher compatibility, improved mask stability under joint STE optimization, and can tolerate up to 62.5% sparsity before collapse [2603.05168]. Empirical results indicate >1.3× throughput gains at near-dense quality. However, N:M sparsity introduces new security vulnerabilities. The Silent Until Sparse (SUS) backdoor attack implants dormant backdoors into potential mask-retained weights, hidden pre-pruning and reliably triggered post-pruning, evading existing detection and remaining robust to fine-tuning [2509.08747]. ## 6. Scaling, Transferability, and Extensions Learnable mask distributions (MaskLLM, MaskPro) scale N:M sparsity to large LLMs and across domains via transfer learning of mask logits, with adaptation and calibration yielding near-lossless compression on downstream tasks [2409.17481, 2506.12876]. CAST demonstrates that differentiable, continuous sparsity-aware training with proportional L1 decay and row/group scaling enables convergence to optimal sparse minima, matching or exceeding prior SOTA under tight retraining budgets [2509.25996]. Hierarchical N:M (HiNM) and permutation strategies—combining vector and fine-grained row-wise sparsity—approach unstructured sparsity accuracy at up to 75% sparsity, facilitated by gyro-permutation and permutation-aware GPU kernels [2407.20496]. Activation-level N:M sparsity, as demonstrated by Amber Pruner and Outstanding-sparse, achieves up to 56% FLOP reduction in prefill for LLMs, approaching ~1.7× speedup on sparsity-aware hardware with negligible to sub-1% accuracy loss after simple plug-and-play error mitigation [2508.02128, 2509.22166]. ## 7. Limitations and Future Directions Current limitations include hardware restrictions to block sizes (primarily 2:4), the complexity of mask generation for large block sizes or transposable constraints, and the need for matched kernel/software support for both weights and activations. Advanced rounding, permutation, and mask optimization methods are being developed for scalability and better accuracy retention at higher sparsity [2505.23949, 2407.20496]. Open directions encompass support for dynamically adaptive N, activation sparsity in both prefill and decode, co-design of network architectures and hardware for end-to-end N:M-aware computation, and integration with quantization, distillation, and continual learning paradigms [2509.25996, 2603.05168, 2508.02128]. --- **Key References:** - [2209.07617] Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask - [2312.07061] MaxQ: Multi-Axis Query for N:M Sparsity Network - [2110.15766] NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM - [2503.01253] NM-SpMM: Accelerating Matrix Multiplication Using N:M Sparsity with GPGPU - [2505.23949] TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks - [2509.22166] Lightweight error mitigation strategies for post-training N:M activation sparsity in LLMs - [2407.20496] Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUs - [2509.25996] CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models - [2409.17481] MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models - [2506.12876] MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on Large Language Models - [2603.05168] Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity - [2509.08747] Silent Until Sparse: Backdoor Attacks on Semi-Structured Sparsity - [2508.02128] Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models - [2206.06662] Learning Best Combination for Efficient N:M Sparsity - [2405.01943] Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models Source: https://www.emergentmind.com/topics/semi-structured-n-m-sparsity