Spense: Hybrid Sparse-Dense Format & Neuroevolution

• Spense is a hybrid sparse–dense format for LLM inference that divides weights into contiguous dense and 2:4 sparse regions to balance speed and accuracy.
• It utilizes one-shot post-training pruning methods, SpenseGPT and SpenseGPT+, to optimize GLU-style MLPs without custom compilers.
• Spense also appears as a misspelling of SPENSER, a grammar-based neuroevolution framework for designing convolutional spiking neural networks.

Searching arXiv for the two relevant senses of “Spense”: the 2026 LLM sparsity format and the 2023 SNN neuroevolution framework. “Spense” denotes two distinct objects in recent arXiv usage. In large-language-model inference, it refers to a hybrid sparse–dense weight format introduced together with the one-shot pruning methods SpenseGPT and SpenseGPT+, designed to relax strict 2:4 semi-structured sparsity while remaining compatible with existing sparse and dense GEMM libraries (Lee et al., 9 Jun 2026). In spiking-neural-network research, “Spense” is also a frequent misspelling of SPENSER, short for SPiking Evolutionary Network StructurEd Representation, a neuroevolutionary framework for automatically designing convolutional spiking neural networks trained with BPTT and surrogate gradients (Branquinho et al., 2023). The two usages are technically unrelated, but both concern automated exploitation of constrained model structure.

1. Terminological scope

The dominant contemporary meaning of Spense is the hybrid sparse–dense format proposed in “SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference,” where each weight matrix is divided into a dense region and a 2:4 sparse region. The design target is practical deployment on modern accelerators using standard vendor libraries rather than custom compilers or activation transformations (Lee et al., 9 Jun 2026).

A second usage arises from the 2023 paper “SPENSER: Towards a NeuroEvolutionary Approach for Convolutional Spiking Neural Networks,” where the framework name is often mistyped as “Spense.” In that context, the referent is not sparsity for transformer inference, but grammar-based neuroevolution for convolutional spiking neural networks, including search over architecture, neuron dynamics, and optimizer hyperparameters (Branquinho et al., 2023).

A common misconception is therefore terminological rather than conceptual: Spense may denote either a sparse-inference format for LLMs or an accidental shortening of SPENSER in SNN literature. The distinction is essential because the former concerns post-training pruning and GEMM execution, whereas the latter concerns evolutionary design of CSNNs.

2. Spense as a hybrid sparse–dense format

Spense was proposed to address a specific limitation of 2:4 semi-structured sparsity. In 2:4 sparsity, for each group of four consecutive weights in a row, at most two are non-zero:

$\| W_{i,4j:4j+4} \|_0 \le 2.$

This pattern is widely supported by modern accelerators and offers up to a theoretical $2\times$ speedup, but its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning, especially for full-scale LLMs (Lee et al., 9 Jun 2026).

Spense relaxes this constraint by splitting a weight matrix into two contiguous regions: one dense and one 2:4 sparse. If $p$ is the fraction kept dense, then the overall density is

$\text{density} = 0.5 + 0.5p,$

and the overall sparsity is

$\text{sparsity} = (1-p)\cdot 0.5.$

The special cases recover the intended continuum: $p=0$ gives standard 2:4 with 50% sparsity, $p=0.25$ gives 37.5% sparsity, and $p=0.5$ gives 25% sparsity (Lee et al., 9 Jun 2026).

The key engineering feature is contiguity. Because the dense and sparse regions are contiguous in memory layout, the sparse block can be executed with cuSPARSELt or an equivalent sparse GEMM library, while the dense block can be executed with cuBLASLt or an equivalent dense GEMM library. This avoids custom compiler support, activation expansion, and complex runtime packing. The paper implements two variants. Output-split Spense partitions along the output dimension and computes separate sparse and dense GEMMs followed by concatenation; it is used particularly for GLU-style MLP up and gate projections. Input-split Spense partitions along the input dimension, computes a sparse GEMM, and then accumulates a dense GEMM contribution; it is used primarily for the MLP down projection (Lee et al., 9 Jun 2026).

This suggests that Spense is best understood not merely as a pruning pattern but as a runtime-compatible layout constraint: it preserves the hardware-favorable local regularity of 2:4 sparsity while relaxing the global requirement that every part of the matrix must be 50% sparse.

3. SpenseGPT and SpenseGPT+

The Spense format does not by itself determine which weights remain dense. That role is played by SpenseGPT and SpenseGPT+, which are one-shot post-training pruning methods that transform a trained dense LLM into Spense format without iterative prune–fine-tune cycles. Both methods use SparseGPT as the core per-layer sparsifier for the 2:4 region, while protecting a designated dense region from pruning (Lee et al., 9 Jun 2026).

The workflow is specialized to GLU-style MLPs with three projections, $W_{\text{up}}$, $W_{\text{gate}}$, and $2\times$0, where

$2\times$1

An intermediate index corresponds jointly to a row of $2\times$2, a row of $2\times$3, and a column of $2\times$4. Because these intermediate indices can be permuted consistently without changing the MLP output, the method first chooses which indices should remain dense, then permutes them into a contiguous tail region, and finally applies SparseGPT only to the remaining part. This permutation step is what makes arbitrary index selection compatible with Spense’s contiguous layout (Lee et al., 9 Jun 2026).

SpenseGPT uses a simple rule: the last $2\times$5 of intermediate indices are designated dense. The rationale is tied to SparseGPT’s left-to-right processing order, because dense columns placed on the right remain available as compensation buffers while earlier columns are pruned. SpenseGPT+ instead ranks intermediate indices with a Wanda-style activation-aware magnitude score aggregated across the gate, up, and down projections, then takes the top $2\times$6 indices before moving them to the tail. The paper reports that which indices are kept dense matters dramatically; on Qwen3-32B, poor dense-index choice can reduce AIME2024 accuracy from 76.67 to 8.89, while the Wanda-style scoring used in SpenseGPT+ performs much better than a SparseGPT-style OBS saliency repurposed for index selection (Lee et al., 9 Jun 2026).

A recurrent misconception is to identify Spense with SpenseGPT. Formally, Spense is the weight format; SpenseGPT and SpenseGPT+ are pruning procedures that produce weights in that format.

4. Empirical behavior in LLM inference

The reported evaluation uses Qwen3-32B and Seed-OSS-36B-Instruct, pruning MLP layers only while keeping attention weights dense. Runtime measurements use NVIDIA B200 GPUs, FP8 precision, and vLLM, with batch size 32 and sequence length 4096 for speed measurement. Calibration uses 128 samples from s1K-1.1 with sequence length 16384. Accuracy is evaluated on AIME2024, GPQA Diamond, IFEval, and LiveCodeBench v6, with averages reported across tasks and each experiment averaged over 3 runs (Lee et al., 9 Jun 2026).

Under these conditions, strict 50% MLP sparsity with standard 2:4 post-training pruning incurs substantial degradation. For Qwen3-32B, the dense baseline average is 71.68, SparseGPT at 50% MLP sparsity gives 60.27, Wanda gives 18.96, and HyperPrune gives 24.22. By contrast, SpenseGPT+ at 25% MLP sparsity reaches 70.76, while SpenseGPT at 25% MLP sparsity reaches 67.21. For Seed-OSS-36B-Instruct, the dense baseline is 73.86, SparseGPT at 50% MLP sparsity gives 65.18, and SpenseGPT+ at 25% MLP sparsity reaches 73.12. The paper therefore presents relaxed effective sparsity, rather than strict 50% sparsity, as the mechanism that preserves quality in one-shot pruning (Lee et al., 9 Jun 2026).

The runtime results are framed as end-to-end decoding rather than isolated kernel benchmarks. Spense achieves up to 1.2× end-to-end decoding speedup on B200 GPUs with FP8 precision, whereas SlideSparse at comparable overall sparsity yields only about 1.04× in the reported setup because activation expansion introduces substantial non-GEMM overhead. The pruning cost is also reported: on a single B200 GPU, SpenseGPT prunes Qwen3-32B in 1.33 h and Seed-OSS-36B in 1.52 h, while SpenseGPT+ requires 1.87 h and 2.15 h respectively; HyperPrune is slower, and ProxSparse is reported as OOM for these scales (Lee et al., 9 Jun 2026).

Two ablations are especially important for interpretation. First, dense-index scoring based on a SparseGPT-style saliency performs poorly compared with the Wanda-style score used by SpenseGPT+. Second, placing dense indices at the back before running SparseGPT is critical: on Qwen3-32B at 25% MLP sparsity, leaving dense indices in place or moving them to the front gives averages of 28.50 and 29.61, whereas moving them to the back gives 70.76. This indicates that the method’s success depends not only on relaxed sparsity, but also on careful alignment between index placement and SparseGPT’s column traversal (Lee et al., 9 Jun 2026).

5. Relation to other sparsity methods and practical limits

Spense is positioned against three alternatives. Relative to strict 2:4 semi-structured sparsity, it keeps the same local per-tuple constraint inside the sparse region but avoids enforcing 50% sparsity everywhere. Relative to PATCH, it does not require specialized Triton compiler support. Relative to SlideSparse, it avoids weight and activation expansion, and thereby avoids the associated non-GEMM overhead that can erase kernel-level gains in end-to-end serving (Lee et al., 9 Jun 2026).

The practical limitations are stated explicitly. Realized speedup depends on batch size, sequence length, hardware backend, precision, and serving framework; sparse GEMM kernels only outperform dense at sufficiently large $2\times$7, so gains may be limited or negative for small batch sizes. SpenseGPT+ depends on calibration data to estimate Hessians and importance scores, so dense-index selection may be suboptimal if deployment data differ substantially. Empirical validation is concentrated on NVIDIA B200 FP8 with vLLM, even though the design is presented as portable in principle to platforms with appropriate sparse and dense GEMM libraries (Lee et al., 9 Jun 2026).

The reported guidance is correspondingly conservative. The recommended starting point is MLPs only, with attention remaining dense. 25% MLP sparsity is presented as a safe operating point, because it nearly matches dense-model accuracy on both evaluated models while still yielding meaningful end-to-end speedup. For implementation simplicity, SpenseGPT is viable; for best accuracy, SpenseGPT+ with Wanda-style dense-index scoring and dense indices placed at the back is preferred (Lee et al., 9 Jun 2026).

6. SPENSER, often mistyped as “Spense”

In the SNN literature, “Spense” often refers inadvertently to SPENSER, a neuroevolutionary framework for automatic design of convolutional spiking neural networks. SPENSER extends DENSER, which combines a high-level genetic-algorithm macrostructure with Structured Grammatical Evolution or Dynamical Structured Grammatical Evolution to specify layer types and hyperparameters through a context-free grammar (Branquinho et al., 2023).

The framework fixes the learning strategy to BPTT with surrogate gradients, but evolves the CSNN architecture and many SNN-specific hyperparameters. Its macrostructure is

$2\times$8

allowing repeated convolutional feature blocks, repeated fully connected classification blocks, a single output block, and a single learning block. The grammar includes convolutional and fully connected layers, pooling, dropout, LIF spiking activation layers, surrogate gradient choice, reset mechanism, and optimizer configuration. Evolved neuron parameters include the decay rate $2\times$9, threshold $p$0, whether these are trainable, the reset mode, and the surrogate gradient choice; the optimizer can be SGD, Adam, or RMSProp with associated hyperparameters (Branquinho et al., 2023).

SPENSER uses the Leaky Integrate-and-Fire neuron model. The continuous-time dynamics are

$p$1

and the discretized update used in the implementation is

$p$2

with spike generation defined by thresholding $p$3. Networks are trained over time with BPTT and surrogate gradients, using a Mean Square Error Spike Count Loss under rate coding at the output. Evolution itself does not optimize weights; it evaluates candidate architectures by training each individual for 1 epoch on EvoTrain and measuring classification accuracy on a Fitness split under a (1+\lambda) ES with $p$4 parent and $p$5 offspring for 200 generations across 5 independent runs (Branquinho et al., 2023).

The reported results are 99.42% test accuracy on MNIST and 91.65% on Fashion-MNIST after additional training of the best evolved networks. Among the ten best individuals, convolutional and fully connected layers each account for 35% of all layers, max pooling for 19%, dropout for 11%, and average pooling for 0%. The preferred reset mechanism is subtractive reset at 63%, the preferred surrogate gradient is ATan at 76%, and Adam appears as optimizer in 70% of the best individuals. The paper presents these results as evidence that grammar-based neuroevolution can discover competitive CSNNs without manual architecture design, though the experiments are limited to MNIST and Fashion-MNIST and do not directly optimize energy or spike count (Branquinho et al., 2023).

Taken together, the two meanings of “Spense” illustrate a purely lexical overlap across unrelated subfields. In one case, the term denotes a hybrid sparse–dense inference format and its associated one-shot pruning algorithms for LLMs; in the other, it denotes, by misspelling, a grammar-based neuroevolution framework for convolutional spiking neural networks.