---
title: 'SpenseGPT: One-shot Pruning for LLMs'
url: https://www.emergentmind.com/topics/spensegpt
type: topic
---

# SpenseGPT: One-shot Pruning for LLMs

Searching arXiv for the cited paper and its key precursor to ground the article in current records.
SpenseGPT is a one-shot post-training pruning method for large language models introduced together with **Spense**, a hybrid sparse–dense weight format for LLM inference [2606.10445]. It targets a specific deployment problem: translating the theoretical appeal of **2:4 semi-structured sparsity** into **real end-to-end decoding speedup on modern GPUs** while preserving model quality. Rather than forcing every local weight group to satisfy strict 50% 2:4 sparsity, SpenseGPT prunes MLP weights into a format that combines a hardware-native 2:4 sparse region with a retained dense region, so inference can be executed using ordinary **sparse GEMM + dense GEMM** calls from existing vendor libraries, without custom compiler support and without activation expansion [2606.10445].

## 1. Problem setting and motivation

SpenseGPT is motivated by the gap between the hardware support for **2:4 semi-structured sparsity** and the difficulty of obtaining useful LLM inference speedups under post-training pruning. For a weight matrix $W \in \mathbb{R}^{N \times K}$ used with PyTorch-style multiplication $XW^T$, 2:4 sparsity imposes
$$
\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.
$$
This format is attractive because hardware can store only nonzeros plus compact metadata and skip zero computations, yielding up to $2\times$ theoretical speedup [2606.10445].

The practical difficulty is that strict 2:4 enforces a **50% sparsity constraint everywhere**. The paper argues that this rigidity often causes non-negligible accuracy degradation under one-shot post-training pruning, especially on harder reasoning tasks. On **Qwen3-32B**, the dense baseline average score is **71.68**, while **SparseGPT at 50% MLP sparsity** drops to **60.27**. On **Seed-OSS-36B-Instruct**, dense is **73.86**, while **SparseGPT** drops to **65.18**; **Wanda/HyperPrune are worse** [2606.10445]. SparseGPT itself is the immediate algorithmic precursor: it is a one-shot, second-order, post-training pruning method for GPT-family LLMs that can also handle 2:4 patterns [2301.00774].

The paper also frames prior relaxed formats as operationally unsatisfactory. **PATCH** uses dense GEMM for some tiles and sparse GEMM for others, but requires **specialized Triton compiler support**, which at the time of writing was only implemented for Ampere GPUs. **SlideSparse** realizes a more relaxed $(2n-2):2n$ pattern by expanding weights into equivalent 2:4 patterns and **expanding activations** at runtime by a factor
$$
\frac{2n-2}{n}.
$$
That activation expansion introduces extra runtime work, so even if GEMM time decreases, non-GEMM overhead can cancel end-to-end gains [2606.10445].

A central systems observation is that sparse tensor core acceleration alone does not guarantee faster serving. The sparse-vs-dense speedup depends strongly on the input batch dimension $M$ in $X \in \mathbb{R}^{M \times K}$; when $M$ is small, sparse GEMM may even be slower than dense GEMM, and only for large $M$ can speedup approach $2\times$. In addition, end-to-end latency includes **non-GEMM overheads**, so any format that requires activation transformation risks losing the practical benefit of sparse kernels [2606.10445].

## 2. Hybrid sparse–dense format

**Spense** is the hybrid sparse–dense format that SpenseGPT produces. Each weight matrix is partitioned into two contiguous regions: a **dense region**, covering $p\%$ of the matrix along one contiguous dimension, and a **2:4 sparse region**, covering the remaining $100-p\%$ [2606.10445]. Conceptually,
$$
W = W_{\text{sparse}} + W_{\text{dense}},
$$
where the two parts are disjoint by construction and occupy contiguous blocks after a possible permutation of intermediate indices. The sparse part satisfies the 2:4 constraint, while the dense part is left unpruned [2606.10445].

The paper implements two execution variants. In **output-split Spense**, the **output dimension** is split into a sparse part and a dense part, so one sparse GEMM and one dense GEMM are run on the same input. In **input-split Spense**, the **input dimension** is split into a sparse part and a dense part; sparse GEMM is run first, then dense GEMM accumulates into the same output tensor. For GLU-style MLPs, the paper uses **output-split** for $W_{\text{up}}$ and $W_{\text{gate}}$, and **input-split** for $W_{\text{down}}$ [2606.10445].

This design reflects a precise tradeoff. Some parameters are kept **dense** to preserve accuracy; the rest are retained in hardware-native **2:4 sparse** form for acceleration. The practical contribution is that Spense is designed to run directly on existing libraries such as **cuSPARSELt** and **cuBLASLt** on NVIDIA GPUs, and **hipSPARSELt** and **hipBLASLt** on AMD GPUs, with **no custom compiler support** and **no activation expansion** [2606.10445].

A plausible implication is that Spense is less a new sparsity pattern than a deployment-oriented representation strategy: it relaxes the effective local sparsity constraint while preserving the execution path expected by existing high-performance sparse and dense GEMM backends.

## 3. One-shot pruning procedure

SpenseGPT is the one-shot post-training pruning algorithm that maps MLP weights into the Spense format [2606.10445]. It builds directly on **SparseGPT**, which reconstructs pruned layers using an **OBS-style second-order approximation** [2301.00774]. For a layer with weight matrix $W \in \mathbb{R}^{N \times K}$ and calibration activations $X \in \mathbb{R}^{M \times K}$, SparseGPT seeks compressed $\widehat{W}$ minimizing
$$
\|XW^T - X\widehat{W}^T\|_F^2.
$$
Using the Hessian approximation
$$
H = X^T X,
$$
SparseGPT applies an Optimal Brain Surgeon style update. When pruning weight $w_m$, the paper gives
$$
\delta_m = - \frac{w_m}{[H^{-1}]_{mm}} H^{-1}_{:,m}, \qquad \varepsilon_m = \frac{w_m^2}{[H^{-1}]_{mm}}.
$$
Here $\delta_m$ is the compensation direction for remaining weights, and $\varepsilon_m$ is the estimated reconstruction error from pruning that weight; weights with smaller $\varepsilon_m$ are preferred for pruning [2606.10445].

The main challenge in SpenseGPT is not only pruning the sparse region but choosing **which part should remain dense**. The method focuses on GLU-style MLPs, where an intermediate index corresponds jointly to a row in $W_{\text{gate}}$, a row in $W_{\text{up}}$, and a column in $W_{\text{down}}$. Its key structural fact is a permutation invariance of the intermediate dimension:
$$
\text{MLP}(X)= \left( X W_{up}^T \odot \sigma(X W_{gate}^T) \right) W_{down}^T
$$
and for any permutation matrix $P$,
$$
\text{MLP}(X) = \left( X(P^T W_{up})^T \odot \sigma(X(P^T W_{gate})^T) \right) (W_{down}P)^T.
$$
This means important intermediate indices may first be selected arbitrarily and then permuted into a contiguous region for efficient Spense execution [2606.10445].

The pruning pipeline for each MLP triplet $(W_{\text{gate}}, W_{\text{up}}, W_{\text{down}})$ is: collect calibration activations; choose a dense-index set $\mathcal{D}$; permute intermediate indices so the selected dense indices become contiguous and are moved to the **back/right**; protect the dense region from pruning; apply SparseGPT only to the complementary region, enforcing 2:4 sparsity there; and store the final result as one dense block plus one 2:4 sparse block [2606.10445]. The “back/right” placement is operationally important because SparseGPT processes columns left-to-right and compensates using future columns.

## 4. Dense-region selection strategies

The paper argues that **dense-region choice is critical** [2606.10445]. With the same amount of dense capacity, poor dense-index selection can collapse quality: in one comparison on Qwen3-32B, **AIME** falls from **76.67** to **8.89** [2606.10445].

SpenseGPT proposes two selection strategies. The first, called **SpenseGPT**, is deliberately simple. Given dense ratio $p$ and number of intermediate indices $N$, it selects the last $p\%$ of indices:
$$
\mathcal{D}_{\text{SpenseGPT}} = \{\lfloor pN \rfloor,\dots,N\}.
$$
The rationale is not that these indices are intrinsically important, but that placing dense columns on the right improves SparseGPT reconstruction because columns are processed left-to-right [2606.10445].

The stronger method, **SpenseGPT+**, estimates dense-index importance using a one-shot **Wanda-style** activation-aware score. The ideal objective is
$$
\min_{\mathcal{D}: |\mathcal{D}| = pN} \big\| X W_{mlp}^T - X \widehat{W}_{mlp}(\mathcal{D})^T \big\|_F^2,
$$
for $mlp \in \{up, gate, down\}$, but direct optimization over $\mathcal{D}$ is combinatorial [2606.10445]. The method therefore uses the elementwise importance
$$
S_{ij} = |W_{ij}| \sqrt{H_{j,j}},
$$
derived from the fact that removing weight $W_{ij}$ changes output by $W_{ij}X_{:,j}$, with squared error contribution proportional to $W_{ij}^2 \|X_{:,j}\|_2^2$ and $\|X_{:,j}\|_2^2 = H_{j,j}$ [2606.10445].

For each intermediate index $i$ in a GLU MLP triplet, the method aggregates scores across the three matrices:
$$
s_i^{mlp} = \frac{1}{K} \sum_{j=1}^{K} |(W_{mlp})_{i,j}| \sqrt{(H_{mlp})_{j,j}},
$$
normalizes each score vector by its mean,
$$
\operatorname{Norm}(s)_i = \frac{s_i}{\frac{1}{N}\sum_{k=1}^{N}s_k+\epsilon},
$$
then forms the final priority score
$$
s_i = \operatorname{Norm}(s^{gate})_i + \operatorname{Norm}(s^{up})_i + \operatorname{Norm}(s^{down})_i,
$$
and selects
$$
\mathcal{D}_{\text{SpenseGPT+}} = \operatorname{TopK}_{pN}(s).
$$
After selection, the chosen indices are permuted into the contiguous back/right dense region and SparseGPT is applied to the complement [2606.10445].

The paper also tests a **SparseGPT-like** saliency score,
$$
S_{ij}^{\text{SGPT}} = \frac{W_{ij}^2}{[H^{-1}]_{j,j}},
$$
aggregated over indices, and finds that it performs very poorly for dense-index selection [2606.10445]. This suggests that SparseGPT saliency is effective inside the full reconstruction procedure but is not suitable as an independent index selector.

## 5. Inference execution and empirical results

Once a layer is converted to Spense, inference is executed as two ordinary GEMMs over disjoint regions. At the operational level,
$$
Y = XW^T = XW_{\text{sparse}}^T + XW_{\text{dense}}^T.
$$
This is implemented as one **sparse GEMM** on the 2:4 region, one **dense GEMM** on the dense region, and accumulation into the output [2606.10445]. For GLU MLPs, $W_{\text{up}}$ and $W_{\text{gate}}$ use output-split execution, while $W_{\text{down}}$ uses input-split execution [2606.10445].

The experimental setup evaluates **Qwen3-32B** and **Seed-OSS-36B-Instruct**, pruning only **MLPs** while leaving attention layers dense. Calibration uses **128 samples** from **s1K-1.1** with sequence length **16384**. Evaluation tasks are **AIME2024**, **GPQA Diamond**, **IFEval**, and **LiveCodeBench v6 (LCB)**. Inference is measured on **NVIDIA B200 GPUs** with **FP8**, using **vLLM**, decoding batch size **32**, and sequence length **4096**; each experiment is averaged over **3 runs** [2606.10445].

The principal quality result is that relaxing strict 2:4 with Spense substantially improves post-training accuracy. On **Qwen3-32B**, average scores are **71.68** for dense, **60.27** for SparseGPT 50%, **65.38** for SpenseGPT 37.5%, **67.79** for SpenseGPT+ 37.5%, **67.21** for SpenseGPT 25%, and **70.76** for SpenseGPT+ 25% [2606.10445]. At **25% MLP sparsity**, SpenseGPT+ nearly matches or exceeds dense on several tasks: **AIME 76.67 vs 75.56**, **GPQA 66.83 vs 65.49**, **IFEval 81.83 vs 85.46**, and **LCB 57.71 vs 60.19** [2606.10445]. On **Seed-OSS-36B-Instruct**, averages are **73.86** for dense, **65.18** for SparseGPT 50%, **70.25** for SpenseGPT 37.5%, **70.35** for SpenseGPT+ 37.5%, **71.69** for SpenseGPT 25%, and **73.12** for SpenseGPT+ 25% [2606.10445].

The headline systems result is **up to $1.2\times$ end-to-end decoding speedup** on **B200 GPUs** with **FP8**, while preserving accuracy [2606.10445]. The paper states that, to the best of the authors’ knowledge, this is the **first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200, while maintaining model quality** [2606.10445].

Pruning time remains in the one-shot regime. On a single B200 GPU, **Qwen3-32B** takes **1.33h** for SparseGPT, **1.33h** for SpenseGPT, and **1.87h** for SpenseGPT+; **Seed-OSS-36B** takes **1.52h**, **1.52h**, and **2.15h**, respectively [2606.10445].

## 6. Ablations, limitations, and significance

The ablations identify dense-index handling as decisive. For Qwen3-32B, a dense-index selector based on SparseGPT-style saliency produces **AIME 8.89**, **GPQA 27.95**, **IFEval 65.19**, **LCB 9.14**, and average **27.79**, whereas **SpenseGPT+** achieves **76.67**, **66.83**, **81.83**, **57.71**, and **70.76** on the same metrics [2606.10445]. Likewise, after selecting dense indices, their placement before pruning is crucial: keeping indices in place gives average **28.50**, moving them to the **front** gives **29.61**, while moving them to the **back** gives **70.76** [2606.10445]. This directly validates the interaction between Spense layout and SparseGPT’s left-to-right compensation procedure.

The paper also clarifies a common misconception in sparse LLM deployment: theoretical sparse-kernel speedups do not necessarily translate to real inference gains. Sparse kernels need enough work to amortize overhead, non-GEMM costs can dominate total latency, and any format that requires activation expansion can lose end-to-end efficiency [2606.10445]. Spense’s systems contribution is therefore inseparable from its pruning method: its value lies in a representation that maps cleanly to existing sparse and dense GEMM libraries with negligible runtime overhead.

Several caveats remain. Realized speedup depends on the quality of the underlying sparse and dense GEMM libraries and may vary with batch size, sequence length, hardware backend, precision, and serving stack. Portability is not fully characterized, since the evaluation is primarily on **B200 + FP8 + vLLM**. SpenseGPT+ depends on calibration data for dense-index selection, so large deployment-distribution shifts may degrade that choice. The experimental scope is limited to **MLPs**, with attention parameters left unchanged. The paper explicitly suggests future work on more robust or task-adaptive dense-index selection strategies [2606.10445].

In the broader pruning literature, SpenseGPT can be understood as a deployment-centered extension of SparseGPT [2301.00774]. Instead of asking only how to reconstruct a pruned matrix under a given sparsity pattern, it asks which subspace should remain dense so that semi-structured sparse tensor cores produce **actual serving gains**. That reframing—hybrid format, contiguous dense subspace, GLU permutation invariance, and one-shot sparse reconstruction—defines its significance within post-training compression for LLM inference [2606.10445].

Source: https://www.emergentmind.com/topics/spensegpt