SpenseGPT: One-shot Pruning for LLMs

• SpenseGPT is a one-shot post-training pruning method for LLMs that converts MLP weights into a hybrid format combining hardware-friendly 2:4 sparse and dense regions.
• It employs a second-order reconstruction strategy inspired by SparseGPT to optimize pruning while balancing accuracy and decoding efficiency using standard GEMM libraries.
• Experimental results on models like Qwen3-32B and Seed-OSS-36B-Instruct demonstrate up to 1.2× decoding speedup with minimal quality loss, validating its deployment benefits.

Searching arXiv for the cited paper and its key precursor to ground the article in current records. SpenseGPT is a one-shot post-training pruning method for LLMs introduced together with Spense, a hybrid sparse–dense weight format for LLM inference (Lee et al., 9 Jun 2026). It targets a specific deployment problem: translating the theoretical appeal of 2:4 semi-structured sparsity into real end-to-end decoding speedup on modern GPUs while preserving model quality. Rather than forcing every local weight group to satisfy strict 50% 2:4 sparsity, SpenseGPT prunes MLP weights into a format that combines a hardware-native 2:4 sparse region with a retained dense region, so inference can be executed using ordinary sparse GEMM + dense GEMM calls from existing vendor libraries, without custom compiler support and without activation expansion (Lee et al., 9 Jun 2026).

1. Problem setting and motivation

SpenseGPT is motivated by the gap between the hardware support for 2:4 semi-structured sparsity and the difficulty of obtaining useful LLM inference speedups under post-training pruning. For a weight matrix $W \in \mathbb{R}^{N \times K}$ used with PyTorch-style multiplication $XW^T$, 2:4 sparsity imposes

$$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$

This format is attractive because hardware can store only nonzeros plus compact metadata and skip zero computations, yielding up to $2\times$ theoretical speedup (Lee et al., 9 Jun 2026).

The practical difficulty is that strict 2:4 enforces a 50% sparsity constraint everywhere. The paper argues that this rigidity often causes non-negligible accuracy degradation under one-shot post-training pruning, especially on harder reasoning tasks. On Qwen3-32B, the dense baseline average score is 71.68, while SparseGPT at 50% MLP sparsity drops to 60.27. On Seed-OSS-36B-Instruct, dense is 73.86, while SparseGPT drops to 65.18; Wanda/HyperPrune are worse (Lee et al., 9 Jun 2026). SparseGPT itself is the immediate algorithmic precursor: it is a one-shot, second-order, post-training pruning method for GPT-family LLMs that can also handle 2:4 patterns (Frantar et al., 2023).

The paper also frames prior relaxed formats as operationally unsatisfactory. PATCH uses dense GEMM for some tiles and sparse GEMM for others, but requires specialized Triton compiler support, which at the time of writing was only implemented for Ampere GPUs. SlideSparse realizes a more relaxed $(2n-2):2n$ pattern by expanding weights into equivalent 2:4 patterns and expanding activations at runtime by a factor

$\frac{2n-2}{n}.$

That activation expansion introduces extra runtime work, so even if GEMM time decreases, non-GEMM overhead can cancel end-to-end gains (Lee et al., 9 Jun 2026).

A central systems observation is that sparse tensor core acceleration alone does not guarantee faster serving. The sparse-vs-dense speedup depends strongly on the input batch dimension $M$ in $X \in \mathbb{R}^{M \times K}$; when $M$ is small, sparse GEMM may even be slower than dense GEMM, and only for large $M$ can speedup approach $XW^T$0. In addition, end-to-end latency includes non-GEMM overheads, so any format that requires activation transformation risks losing the practical benefit of sparse kernels (Lee et al., 9 Jun 2026).

2. Hybrid sparse–dense format

Spense is the hybrid sparse–dense format that SpenseGPT produces. Each weight matrix is partitioned into two contiguous regions: a dense region, covering $XW^T$1 of the matrix along one contiguous dimension, and a 2:4 sparse region, covering the remaining $XW^T$2 (Lee et al., 9 Jun 2026). Conceptually,

$XW^T$3

where the two parts are disjoint by construction and occupy contiguous blocks after a possible permutation of intermediate indices. The sparse part satisfies the 2:4 constraint, while the dense part is left unpruned (Lee et al., 9 Jun 2026).

The paper implements two execution variants. In output-split Spense, the output dimension is split into a sparse part and a dense part, so one sparse GEMM and one dense GEMM are run on the same input. In input-split Spense, the input dimension is split into a sparse part and a dense part; sparse GEMM is run first, then dense GEMM accumulates into the same output tensor. For GLU-style MLPs, the paper uses output-split for $XW^T$4 and $XW^T$5, and input-split for $XW^T$6 (Lee et al., 9 Jun 2026).

This design reflects a precise tradeoff. Some parameters are kept dense to preserve accuracy; the rest are retained in hardware-native 2:4 sparse form for acceleration. The practical contribution is that Spense is designed to run directly on existing libraries such as cuSPARSELt and cuBLASLt on NVIDIA GPUs, and hipSPARSELt and hipBLASLt on AMD GPUs, with no custom compiler support and no activation expansion (Lee et al., 9 Jun 2026).

A plausible implication is that Spense is less a new sparsity pattern than a deployment-oriented representation strategy: it relaxes the effective local sparsity constraint while preserving the execution path expected by existing high-performance sparse and dense GEMM backends.

3. One-shot pruning procedure

SpenseGPT is the one-shot post-training pruning algorithm that maps MLP weights into the Spense format (Lee et al., 9 Jun 2026). It builds directly on SparseGPT, which reconstructs pruned layers using an OBS-style second-order approximation (Frantar et al., 2023). For a layer with weight matrix $XW^T$7 and calibration activations $XW^T$8, SparseGPT seeks compressed $XW^T$9 minimizing

$$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$0

Using the Hessian approximation

$$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$1

SparseGPT applies an Optimal Brain Surgeon style update. When pruning weight $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$2, the paper gives

$$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$3

Here $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$4 is the compensation direction for remaining weights, and $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$5 is the estimated reconstruction error from pruning that weight; weights with smaller $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$6 are preferred for pruning (Lee et al., 9 Jun 2026).

The main challenge in SpenseGPT is not only pruning the sparse region but choosing which part should remain dense. The method focuses on GLU-style MLPs, where an intermediate index corresponds jointly to a row in $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$7, a row in $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$8, and a column in $$\| W_{i,\,4j:4j+4} \|_0 \le 2, \quad \forall i \le N,\; j \le K/4.$$9. Its key structural fact is a permutation invariance of the intermediate dimension:

$2\times$0

and for any permutation matrix $2\times$1,

$2\times$2

This means important intermediate indices may first be selected arbitrarily and then permuted into a contiguous region for efficient Spense execution (Lee et al., 9 Jun 2026).

The pruning pipeline for each MLP triplet $2\times$3 is: collect calibration activations; choose a dense-index set $2\times$4; permute intermediate indices so the selected dense indices become contiguous and are moved to the back/right; protect the dense region from pruning; apply SparseGPT only to the complementary region, enforcing 2:4 sparsity there; and store the final result as one dense block plus one 2:4 sparse block (Lee et al., 9 Jun 2026). The “back/right” placement is operationally important because SparseGPT processes columns left-to-right and compensates using future columns.

4. Dense-region selection strategies

The paper argues that dense-region choice is critical (Lee et al., 9 Jun 2026). With the same amount of dense capacity, poor dense-index selection can collapse quality: in one comparison on Qwen3-32B, AIME falls from 76.67 to 8.89 (Lee et al., 9 Jun 2026).

SpenseGPT proposes two selection strategies. The first, called SpenseGPT, is deliberately simple. Given dense ratio $2\times$5 and number of intermediate indices $2\times$6, it selects the last $2\times$7 of indices:

$2\times$8

The rationale is not that these indices are intrinsically important, but that placing dense columns on the right improves SparseGPT reconstruction because columns are processed left-to-right (Lee et al., 9 Jun 2026).

The stronger method, SpenseGPT+, estimates dense-index importance using a one-shot Wanda-style activation-aware score. The ideal objective is

$2\times$9

for $(2n-2):2n$0, but direct optimization over $(2n-2):2n$1 is combinatorial (Lee et al., 9 Jun 2026). The method therefore uses the elementwise importance

$(2n-2):2n$2

derived from the fact that removing weight $(2n-2):2n$3 changes output by $(2n-2):2n$4, with squared error contribution proportional to $(2n-2):2n$5 and $(2n-2):2n$6 (Lee et al., 9 Jun 2026).

For each intermediate index $(2n-2):2n$7 in a GLU MLP triplet, the method aggregates scores across the three matrices:

$(2n-2):2n$8

normalizes each score vector by its mean,

$(2n-2):2n$9

then forms the final priority score

$\frac{2n-2}{n}.$0

and selects

$\frac{2n-2}{n}.$1

After selection, the chosen indices are permuted into the contiguous back/right dense region and SparseGPT is applied to the complement (Lee et al., 9 Jun 2026).

The paper also tests a SparseGPT-like saliency score,

$\frac{2n-2}{n}.$2

aggregated over indices, and finds that it performs very poorly for dense-index selection (Lee et al., 9 Jun 2026). This suggests that SparseGPT saliency is effective inside the full reconstruction procedure but is not suitable as an independent index selector.

5. Inference execution and empirical results

Once a layer is converted to Spense, inference is executed as two ordinary GEMMs over disjoint regions. At the operational level,

$\frac{2n-2}{n}.$3

This is implemented as one sparse GEMM on the 2:4 region, one dense GEMM on the dense region, and accumulation into the output (Lee et al., 9 Jun 2026). For GLU MLPs, $\frac{2n-2}{n}.$4 and $\frac{2n-2}{n}.$5 use output-split execution, while $\frac{2n-2}{n}.$6 uses input-split execution (Lee et al., 9 Jun 2026).

The experimental setup evaluates Qwen3-32B and Seed-OSS-36B-Instruct, pruning only MLPs while leaving attention layers dense. Calibration uses 128 samples from s1K-1.1 with sequence length 16384. Evaluation tasks are AIME2024, GPQA Diamond, IFEval, and LiveCodeBench v6 (LCB). Inference is measured on NVIDIA B200 GPUs with FP8, using vLLM, decoding batch size 32, and sequence length 4096; each experiment is averaged over 3 runs (Lee et al., 9 Jun 2026).

The principal quality result is that relaxing strict 2:4 with Spense substantially improves post-training accuracy. On Qwen3-32B, average scores are 71.68 for dense, 60.27 for SparseGPT 50%, 65.38 for SpenseGPT 37.5%, 67.79 for SpenseGPT+ 37.5%, 67.21 for SpenseGPT 25%, and 70.76 for SpenseGPT+ 25% (Lee et al., 9 Jun 2026). At 25% MLP sparsity, SpenseGPT+ nearly matches or exceeds dense on several tasks: AIME 76.67 vs 75.56, GPQA 66.83 vs 65.49, IFEval 81.83 vs 85.46, and LCB 57.71 vs 60.19 (Lee et al., 9 Jun 2026). On Seed-OSS-36B-Instruct, averages are 73.86 for dense, 65.18 for SparseGPT 50%, 70.25 for SpenseGPT 37.5%, 70.35 for SpenseGPT+ 37.5%, 71.69 for SpenseGPT 25%, and 73.12 for SpenseGPT+ 25% (Lee et al., 9 Jun 2026).

The headline systems result is up to $\frac{2n-2}{n}.$7 end-to-end decoding speedup on B200 GPUs with FP8, while preserving accuracy (Lee et al., 9 Jun 2026). The paper states that, to the best of the authors’ knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200, while maintaining model quality (Lee et al., 9 Jun 2026).

Pruning time remains in the one-shot regime. On a single B200 GPU, Qwen3-32B takes 1.33h for SparseGPT, 1.33h for SpenseGPT, and 1.87h for SpenseGPT+; Seed-OSS-36B takes 1.52h, 1.52h, and 2.15h, respectively (Lee et al., 9 Jun 2026).

6. Ablations, limitations, and significance

The ablations identify dense-index handling as decisive. For Qwen3-32B, a dense-index selector based on SparseGPT-style saliency produces AIME 8.89, GPQA 27.95, IFEval 65.19, LCB 9.14, and average 27.79, whereas SpenseGPT+ achieves 76.67, 66.83, 81.83, 57.71, and 70.76 on the same metrics (Lee et al., 9 Jun 2026). Likewise, after selecting dense indices, their placement before pruning is crucial: keeping indices in place gives average 28.50, moving them to the front gives 29.61, while moving them to the back gives 70.76 (Lee et al., 9 Jun 2026). This directly validates the interaction between Spense layout and SparseGPT’s left-to-right compensation procedure.

The paper also clarifies a common misconception in sparse LLM deployment: theoretical sparse-kernel speedups do not necessarily translate to real inference gains. Sparse kernels need enough work to amortize overhead, non-GEMM costs can dominate total latency, and any format that requires activation expansion can lose end-to-end efficiency (Lee et al., 9 Jun 2026). Spense’s systems contribution is therefore inseparable from its pruning method: its value lies in a representation that maps cleanly to existing sparse and dense GEMM libraries with negligible runtime overhead.

Several caveats remain. Realized speedup depends on the quality of the underlying sparse and dense GEMM libraries and may vary with batch size, sequence length, hardware backend, precision, and serving stack. Portability is not fully characterized, since the evaluation is primarily on B200 + FP8 + vLLM. SpenseGPT+ depends on calibration data for dense-index selection, so large deployment-distribution shifts may degrade that choice. The experimental scope is limited to MLPs, with attention parameters left unchanged. The paper explicitly suggests future work on more robust or task-adaptive dense-index selection strategies (Lee et al., 9 Jun 2026).

In the broader pruning literature, SpenseGPT can be understood as a deployment-centered extension of SparseGPT (Frantar et al., 2023). Instead of asking only how to reconstruct a pruned matrix under a given sparsity pattern, it asks which subspace should remain dense so that semi-structured sparse tensor cores produce actual serving gains. That reframing—hybrid format, contiguous dense subspace, GLU permutation invariance, and one-shot sparse reconstruction—defines its significance within post-training compression for LLM inference (Lee et al., 9 Jun 2026).