SliceGPT: Transformer Compression

• SliceGPT is a post-training compression method for transformers that reduces hidden dimensions by slicing weight matrices based on PCA-calibrated activations.
• It leverages computational invariance under orthogonal rotations to realign activations, facilitating the removal of low-importance dimensions without drastic performance loss.
• Empirical results demonstrate up to 25% parameter reduction with maintained accuracy and significant throughput improvements on standard dense kernels.

Searching arXiv for SliceGPT and closely related papers to ground the article in the cited literature. SliceGPT is a post-training compression method for transformer LLMs that reduces the model’s hidden or embedding dimension by deleting rows and columns from weight matrices after rewriting the network in a more compressible basis. Rather than introducing unstructured sparsity, it produces a smaller dense model whose inference can use standard dense kernels. Its central mechanism is a transformer-specific computational invariance under orthogonal rotations, combined with PCA on calibration activations to align low-importance directions with coordinates that can be removed with limited functional loss (Ashkboos et al., 2024).

1. Conceptual basis and definition

SliceGPT is a sparsification scheme in which each weight matrix is replaced by a smaller dense matrix, thereby shrinking the embedding dimension of the network. In a transformer block, it operates on the linear maps associated with attention and feedforward sublayers by deleting rows of input-side matrices, columns of output-side matrices, corresponding rows in the embeddings, and the aligned dimensions in the residual pathway. The result is a model whose representation dimension is reduced from $D$ to a smaller $D_{\text{small}}$, while preserving the standard dense-matrix execution model (Ashkboos et al., 2024).

The method differs from unstructured pruning in both representation and deployment consequences. Sparse methods such as 2:4 sparsity can reduce parameter counts inside matrices, but their practical speedups depend strongly on sparse-kernel support. SliceGPT instead removes entire hidden dimensions, so the compressed network is materially rewritten as a smaller dense transformer. This is why the method is positioned as a structured compression scheme with direct memory and throughput implications rather than as a sparsity pattern that is efficient mainly on paper (Ashkboos et al., 2024).

A common misunderstanding is to treat SliceGPT as layer deletion. That description is inaccurate. The method slices within every layer by removing dimensions from the residual-stream interface and the matrices coupled to it; it does not delete full transformer blocks. Later comparisons against methods such as ShortGPT make this distinction especially salient, because whole-layer deletion and intra-layer slicing have markedly different effects on perplexity and retained functionality (Dumitru et al., 2024).

2. Computational invariance and the linear-algebraic mechanism

The theoretical foundation of SliceGPT is a computational invariance of transformer networks under orthogonal basis changes. For an orthogonal matrix $\mathbf{R}$ satisfying $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$, the paper states the key RMSNorm relation as

$$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$

Because transformer blocks are built from linear maps, RMSNorm, and residual connections, this invariance allows hidden states to be rotated before normalization and rotated back afterward without changing the network function. The rotations can then be absorbed into adjacent weights. In simplified form, for block $\ell$,

$$\tilde{\mathbf{W}}_{\text{in}}^\ell = \mathbf{R}_\ell^\top \mathbf{W}_{\text{in}}^\ell,\qquad \tilde{\mathbf{W}}_{\text{out}}^\ell = \mathbf{W}_{\text{out}}^\ell \mathbf{R}_\ell,$$

with analogous transformations for biases, embeddings, and the LM head. When different blocks use different rotations, the residual path is adjusted by inserting

$\mathbf{R}_{\ell-1}^\top \mathbf{R}_\ell.$

This gives a theorem-level equivalence between the original network and its rotated parameterization (Ashkboos et al., 2024).

Once the network is represented in a rotated basis, SliceGPT uses PCA on calibration activations to identify a low-dimensional signal subspace. For each block $\ell$, it collects calibration activations

$\mathbf{X}_\ell = \{\mathbf{x}_{\ell,i}\}_i,$

forms

$D_{\text{small}}$0

and chooses $D_{\text{small}}$1 as the eigenvectors of $D_{\text{small}}$2, sorted by decreasing eigenvalue. The retained subspace is represented by a selection matrix $D_{\text{small}}$3, yielding the PCA reconstruction objective

$D_{\text{small}}$4

In this basis, the lowest-variance principal directions are explicitly moved to the trailing coordinates and can be sliced away by deleting corresponding rows and columns from the transformed matrices (Ashkboos et al., 2024).

This construction explains why SliceGPT is both structured and basis-dependent. The method does not search for isolated small-magnitude weights; it first changes coordinates so that the least important directions of the hidden space occupy removable axes. A later refinement summarized this geometry as projecting a data matrix $D_{\text{small}}$5 with eigenvectors $D_{\text{small}}$6 and a deletion matrix $D_{\text{small}}$7,

$D_{\text{small}}$8

while minimizing reconstruction error $D_{\text{small}}$9 (Dumitru et al., 2024).

3. Calibration procedure, implementation details, and recovery fine-tuning

The algorithmic pipeline begins by converting LayerNorm-connected models into an RMSNorm-equivalent form by absorbing the LayerNorm linear parts into adjacent matrices. The model is then run on a calibration dataset, blockwise PCA is performed on collected activations, and the network is rewritten in the corresponding rotated bases before the trailing principal directions are removed. The main calibration datasets reported are the WikiText-2 training dataset and the Alpaca training dataset, with a typical calibration set size of 1024 and sequence length 2048. For numerical stability, the PCA eigenvectors are computed in double precision; the paper notes that FP32 PCA can noticeably hurt accuracy for larger models (Ashkboos et al., 2024).

The method was evaluated on OPT models from 125M through 66B, on the Llama and Llama 2 family at 7B, 13B, and 70B, and on Phi-2. For Llama 70B, PCA computation was reported on a single H100 80GB in about 3.5 hours. Throughput and inference measurements were reported on Quadro RTX6000 24GB, A100 40GB, and H100 80GB hardware. Evaluation used WikiText-2 perplexity and zero-shot tasks including PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c through LM Evaluation Harness (Ashkboos et al., 2024).

Quantitatively, SliceGPT was reported to remove up to 25% of total model parameters, including embeddings, for Llama 70B, OPT 66B, and Phi-2 while maintaining 99%, 99%, and 90% zero-shot task performance of the dense model, respectively. Representative WikiText-2 perplexity results include Llama 70B with dense PPL $\mathbf{R}$0, SparseGPT 2:4 at $\mathbf{R}$1, and SliceGPT 25% at $\mathbf{R}$2; for OPT 66B, dense PPL was $\mathbf{R}$3, SparseGPT 2:4 was $\mathbf{R}$4, and SliceGPT 25% was $\mathbf{R}$5. On H100 80GB, Llama 70B at 25% slicing reached $\mathbf{R}$6 throughput, and at 50% slicing reached $\mathbf{R}$7 throughput on one GPU, or $\mathbf{R}$8 for a fixed 2-GPU budget versus dense. For generating a single token on RTX6000 24GB, Llama 70B dropped from 1764 GPUms dense to 1075 GPUms at 25% slicing, reducing compute to 64%; on A100 40GB, the compute was reduced to 66% (Ashkboos et al., 2024).

Although SliceGPT is fundamentally post-training, the original study also examined LoRA-based recovery fine-tuning. Recovery fine-tuning helped substantially for Llama and Phi-2, especially with Alpaca, but added extra training cost. A representative result is Llama 70B at 30% slicing with Alpaca recovery fine-tuning reaching 74.3% mean zero-shot accuracy versus 76.6% dense. OPT models did not benefit meaningfully from the reported recovery fine-tuning configurations. This establishes an important practical nuance: SliceGPT can operate one-shot and post hoc, but the highest downstream accuracy in some settings comes from combining slicing with limited recovery training (Ashkboos et al., 2024).

4. Empirical behavior, trade-offs, and architectural sensitivities

SliceGPT’s main advantage is that it reduces both compute and memory by shrinking the residual-stream dimension itself. Since the compressed model remains dense, inference uses ordinary dense kernels and does not require special sparse implementations. This differentiates it from both unstructured pruning and low-rank decomposition. Low-rank methods replace a matrix with a product of smaller matrices, while SliceGPT keeps a single smaller dense matrix and therefore avoids factorization overhead at inference time (Ashkboos et al., 2024).

The empirical trade-offs are governed by slicing rate, calibration quality, model family, and architecture. More slicing gives larger speed and memory gains but also larger accuracy and perplexity degradation. The paper reports results at 10%, 20%, 25%, 30%, and even 50% slicing; 50% yields large throughput gains but substantially worse perplexity. Larger models tended to tolerate slicing better than smaller ones. OPT models were generally more compressible than Llama models, and Llama’s embedding or activation spectrum was described as more tightly compressed, making slicing harder (Ashkboos et al., 2024).

Calibration sensitivity is a recurrent theme. Larger calibration sets and longer sequences generally improved results, while too-small calibration sets degraded perplexity. The reliance on RMSNorm invariance also imposes architectural constraints: the invariance argument is cleanest for RMSNorm-connected transformers, and LayerNorm models must first be converted into an RMSNorm-equivalent form. Residual connections further require inter-block transforms when different rotations are used in different layers (Ashkboos et al., 2024).

Later work sharpened the interpretation of these behaviors by focusing on the representation dimension itself. In a mechanistic analysis of SliceGPT-pruned LLMs, the representation dimension $\mathbf{R}$9—also described as the embedding dimension, model dimension, or residual stream dimension—was identified as the common structural quantity threading through attention, MLP, normalization, and output behavior. With sparsity $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$0, the post-pruning representation dimension is $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$1, and the paper argued that this dimension dominates the linear transformations, model predictions, and final performance of the pruned model (Xu et al., 6 Mar 2025).

That study proposed analytical relations linking SliceGPT sparsity to performance. For perplexity, it states

$\mathbf{R}^\top\mathbf{R}=\mathbf{I}$2

and for multiple-choice accuracy it reports the empirical relation

$\mathbf{R}^\top\mathbf{R}=\mathbf{I}$3

These formulas were validated on Llama-3-8B-Instruct and Phi-3-mini-4k-Instruct using WikiText2, ARC-e, ARC-c, WinoGrande, and PIQA, with low reported RMSEs for the fitted linear forms. The significance of this analysis is not that SliceGPT becomes architecture-agnostic, but that the dominant compression variable can be interpreted as a reduction of the residual-stream interface shared across the transformer (Xu et al., 6 Mar 2025).

5. Extensions and derivative uses

A direct refinement of SliceGPT replaces its uniform layerwise slice ratio with a dynamic allocation rule. In the original SliceGPT formulation as summarized by that work, the same slicing percentage is applied to every layer. The dynamic extension argues that this constant-slice-everywhere assumption is too blunt and introduces a layer-adaptive policy driven by a Layer Redundancy score defined as the cosine similarity between a layer’s input and output,

$\mathbf{R}^\top\mathbf{R}=\mathbf{I}$4

Higher cosine similarity indicates a more redundant layer and therefore a layer that can be sliced more aggressively. With target average slice percentage $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$5 and slice base $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$6, the method computes

$\mathbf{R}^\top\mathbf{R}=\mathbf{I}$7

so that the average slice rate remains $\mathbf{R}^\top\mathbf{R}=\mathbf{I}$8 while the per-layer allocation varies. On Llama3-8B and Mistral-7B, evaluated on PiQA, HellaSwag, WinoGrande, ARC-Easy, and WikiText-v2 with 1000 samples per dataset, this dynamic policy consistently improved or maintained accuracy and reduced perplexity relative to constant slicing at 30%, 35%, and 40% pruning. For example, on Llama3-8B at 40% pruning, average accuracy improved from 44.3% to 45.8% and WikiText-v2 perplexity improved from 20.69 to 19.11 (Dumitru et al., 2024).

SliceGPT has also been used as an enabling preprocessing step for symmetry-aware storage. A separate line of work argues that SliceGPT-pruned transformers expose additional rotational symmetries because of the altered skip-connection structure, and that these symmetries can be exploited with bits-back coding. The coding argument distinguishes the cost of encoding a specific parameter configuration,

$\mathbf{R}^\top\mathbf{R}=\mathbf{I}$9

from the cost of encoding its equivalence class under symmetry,

$$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$0

For a SliceGPT-pruned transformer block with 6 sliced weight matrices, 2 skip-connection matrices, precision $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$1, slicing rate $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$2, and remaining rate $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$3, the approximate fractional reduction is derived as

$$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$4

Empirically, this yielded an additional 3–5% reduction in total bit usage on top of SliceGPT across SliceGPT-pruned OPT-1.3B, OPT-2.7B, OPT-6.7B, OPT-13B, and Llama-2-7B, with negligible performance impact at finite precision, mainly float16 (He et al., 2024).

These extensions illustrate two distinct trajectories of SliceGPT research. One trajectory modifies the pruning policy while retaining the same structural principle of slicing hidden dimensions. The other treats SliceGPT as an upstream reparameterization whose altered symmetry structure can support downstream compression beyond pruning alone.

6. Comparative position, misconceptions, and limitations

SliceGPT is often treated as a strong baseline for width pruning, but several later studies position it differently depending on the objective. When the target is deployment-oriented post-training compression with real dense-kernel speedups, SliceGPT remains distinctive because it directly rewrites the model into smaller dense matrices. When the target is maximum benchmark retention under additional training, other methods have been reported to outperform it in specific settings (Ashkboos et al., 2024).

Adapt-Pruner, for example, compares against SliceGPT on LLaMA-3.1-8B and related models and argues that layer-wise adaptive pruning plus recovery training better exploits decoder-layer importance skew. At 20%, 40%, and 60% sparsity on LLaMA-3.1-8B, it reports SliceGPT average accuracies of 48.69, 37.09, and 34.48 versus 56.98, 44.13, and 36.44 for Adapt-Pruner, with corresponding WikiText2 perplexities of 19.61, 85.60, and 218.96 for SliceGPT versus 14.54, 33.75, and 119.95 for Adapt-Pruner. The same paper summarizes its advantage over LLM-Pruner, FLAP, and SliceGPT as about 1%–7% accuracy on commonsense benchmarks (Pan et al., 5 Feb 2025).

DeltaLLM places SliceGPT in yet another comparative frame. Rather than deleting structure, DeltaLLM compresses by sharing weights across blocks and representing inter-block differences with low-rank deltas trained using knowledge distillation and progressive module replacement. In its comparisons, SliceGPT-derived SlicedPhi and SlicedLlama serve as baselines that received LoRA recovery fine-tuning, whereas the DeltaLLM models did not. For instance, DELTALLAMA 2.52B at 21% compression is reported with average zero-shot accuracy 0.51 versus 0.33 for SlicedLlama 2.56B at 20% compression; DELTAPHI 3.35B at 12% compression is reported with average 0.59 versus 0.56 for SlicedPhi 3.32B at 12% compression (Mikaelyan et al., 30 Jan 2025).

These comparisons should not be read as refutations of SliceGPT’s original contribution. They instead delimit its niche. SliceGPT is not primarily a training-based distillation method, not a layer-removal method, and not a pure storage-format innovation. Its principal contribution is a theoretically grounded post-training transformation that makes hidden-dimension reduction possible inside transformer layers while retaining dense execution (Ashkboos et al., 2024).

Its limitations follow directly from that design. Accuracy degrades as slicing becomes more aggressive; calibration quality matters; the cleanest theoretical formulation depends on RMSNorm-connected architectures; different model families exhibit different compressibility; and the best retained accuracy may require recovery fine-tuning. At the same time, whole-layer deletion can damage generation quality much more severely: in a direct comparison on Llama3-8B, ShortGPT produced perplexities around $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$5, $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$6, and $$\mathrm{RMSNorm}(\mathbf{X}\mathbf{R})\mathbf{R}^\top = \mathrm{RMSNorm}(\mathbf{X}).$$7 at three pruning levels where dynamic SliceGPT-based slicing remained in the low tens, around 13, 15, and 19 (Dumitru et al., 2024).

Within the broader model-compression literature, SliceGPT therefore occupies a specific position. It established that a transformer can be compressed by slicing away dimensions inside every layer through an orthogonally invariant change of basis, producing a smaller dense model with real inference savings. Subsequent work has refined its allocation rule, analyzed its dominant structural variable, exploited the symmetries of its pruned parameterization, and benchmarked it against adaptive-pruning and weight-sharing alternatives. The resulting picture is of a method that is both foundational and bounded: foundational because it introduced a new invariance-based route to transformer compression, and bounded because later methods can outperform it when they are allowed different forms of adaptation, retraining, or storage-side coding (Ashkboos et al., 2024).