SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Abstract: Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

• The paper presents a one-shot hybrid sparse-dense pruning strategy that retains near-baseline accuracy while significantly reducing inference latency.
• It integrates standard 2:4 sparsity with dense index buffering to exploit hardware-optimized GEMM libraries, ensuring theoretical speedups materialize.
• Empirical results on Qwen3-32B and Seed-OSS-36B show up to 1.2× decoding speed improvement at 25% MLP sparsity with minimal performance drop.

SpenseGPT: A Practical Hybrid Sparse-Dense Approach for One-Shot LLM Pruning and Inference

Motivation and Background

Efficient LLM inference increasingly relies on semi-structured sparsity, notably the 2:4 format, supported natively by modern accelerators such as NVIDIA B200 GPUs. The 2:4 sparsity mandates pruning exactly two out of every four consecutive weights in a matrix, thereby enabling hardware-level computation skip and offering a theoretical speedup of up to $2\times$. However, enforcing this strict 50% sparsity routinely leads to significant degradation of model performance, particularly on complex reasoning and instruction-following tasks, unless extensive retraining is performed.

Prior attempts to relax the rigidity of 2:4 sparsity, like PATCH and SlideSparse, either demand specialized compiler infrastructure or introduce substantial runtime overhead—markedly diminishing their realized speedups in end-to-end LLM deployments. These limitations impede practical utility despite theoretical gains.

Spense Format: Hybrid Sparse-Dense Decomposition

Spense is proposed as a pragmatic solution to the sparsity-quality-speedup trilemma. Unlike prior relaxed-sparsity schemes, Spense splits each weight matrix into two contiguous regions: a standard 2:4 sparse region and a dense region. This design is crucially compatible with high-performance GEMM libraries (cuSPARSELt for sparse, cuBLAS for dense), ensuring both hardware acceleration and negligible auxiliary overhead, thus bypassing input activation expansion and compiler dependency.

Figure 1: Sparse GEMM speedup versus dense GEMM for various $M$ values on Qwen3-32B running on a B200.

Empirical profiling confirms that, for sufficiently large $M$ (input batch size), sparse GEMM realizes its theoretical $2\times$ speedup, but falls below dense GEMM performance for small $M$. Spense leverages this hardware envelope, ensuring that only regions compatible with optimal kernel conditions are sparsified while dense regions buffer accuracy loss.

Figure 2: Latency breakdown of GEMM and non-GEMM operations at $M=65536$ for the gate/up projection of Qwen3-32B on a B200. PATCH is omitted due to Ampere GPU specificity.

Crucially, Spense avoids non-GEMM overhead that plagues SlideSparse, thereby translating GEMM speedup directly into real-world latency reduction.

SpenseGPT and SpenseGPT+: One-Shot Pruning and Dense-Index Optimization

Building atop Spense, SpenseGPT introduces one-shot post-training pruning strategies for hybrid sparse-dense LLMs. The method exploits the invariance in GLU-style MLPs where intermediate indices (rows/columns) can be permuted without altering the semantics, allowing informed selection of which indices to retain dense.

Figure 3: Implementation of output-split and input-split Spense.

SpenseGPT employs a straightforward approach: contiguous dense indices are preserved—preferably at the end of the matrix, aligning with SparseGPT’s left-to-right compensation mechanism, whereby dense columns provide a reconstruction buffer.

SpenseGPT+ refines this by quantifying the reconstruction importance of each intermediate index using Wanda-inspired activation-weight product scores, normalized across projections. Dense indices are greedily selected based on this estimated loss aggregator, further raising accuracy retention.

Figure 4: When pruning, we select which intermediate indices to keep dense. When deploying, we rearrange the indices for kernel efficiency.

Placement of selected dense indices, particularly rear-loading before pruning, is shown to markedly enhance reconstruction efficacy, as confirmed experimentally.

Empirical Evaluation

Comprehensive evaluations are performed on the Qwen3-32B and Seed-OSS-36B LLMs, targeting MLP-only pruning. The benchmarks include AIME, GPQA, IFEval, and LiveCodeBench. Calibration utilizes 128 samples drawn from s1K-1.1, using vllm as the decoding framework with B200 GPUs operating in FP8 precision.

Accuracy results highlight the necessity of hybrid sparse-dense formatting: strict 2:4 pruning (SparseGPT at 50% sparsity) slashes average scores by over 15% across tasks. SpenseGPT at 25% sparsity narrows the gap—on Seed-OSS-36B, scores reach 71.69 (vs. 73.86 dense baseline), and SpenseGPT+ restores average performance almost identically to the unpruned model. Notably, the choice and layout of dense indices impact accuracy dramatically—a misplaced selection can drop scores to single digits.

Spense delivers up to $1.2\times$ end-to-end decoding speedup at 25% MLP sparsity, outperforming SlideSparse (which stagnates at $1.04\times$ due to activation expansion overhead).

Figure 5: End-to-end decoding speedup over the dense baseline, targeting 25\% sparsity. Spense outperforms SlideSparse on B200 and vllm.

Pruning time is competitive; SpenseGPT variants prune 32B-36B models within ~2 hours on a single B200 GPU, while memory-intensive baselines (e.g., ProxSparse) may fail on comparable hardware.

Implications and Future Prospects

The Spense format, via one-shot post-training pruning, advances practical LLM deployment by guaranteeing compatibility with existing accelerator libraries and achieving tangible gains in latency while preserving accuracy. The introduction of activation-aware dense-index selection substantiates the criticality of index importance scoring and index placement in hybrid pruning workflows.

This work exposes foundational directions in post-training quantization and sparsity: further portability studies across hardware backends, robustness of dense-index selection to calibration drift, and advanced adaptive pruning regimes remain open research frontiers. The potential for task-adaptive index scoring and dynamic region partitioning could refine both theoretical and operational aspects of sparse-dense hybrid inference.

Conclusion

SpenseGPT concretely demonstrates that hybrid sparse-dense matrix decomposition, paired with judicious dense-index selection and one-shot pruning, enables real end-to-end speedups for LLM inference on modern accelerators without sacrificing model quality. The synergy with existing GEMM libraries and the scalability of pruning highlight the practical viability of semi-structured sparsity relaxation. This paradigm is poised to inform future methods for efficient, large-scale LLM deployment and serve as a foundation for subsequent research in hardware-aware model compression.