---
title: 'COMPACT: Optimized Transformer Pruning'
url: https://www.emergentmind.com/topics/compact-fe023ef7-94db-40a5-b5ed-ddfb006e804e
type: topic
---

# COMPACT: Optimized Transformer Pruning

COMPACT, short for “Common-token Optimized Model Pruning Across Channels and Tokens,” is a pruning method for decoder-only transformers that jointly prunes rare vocabulary and feed-forward network (FFN) intermediate channels in order to reduce memory, latency, and serving cost while preserving a standard transformer architecture [2509.06836]. The method combines two operations: removing the rarest vocabulary entries from the embedding and unembedding matrices, and pruning FFN channels using common-token-weighted activations so that channel importance is aligned with the post-pruning token distribution. It is designed to be training-free, scale-adaptive, and deployment-friendly, and experiments across Qwen, LLaMA, and Gemma families from 0.5B to 70B report state-of-the-art downstream task performance at similar or higher pruning ratios together with substantial reductions in parameters, GPU memory, and end-to-end latency [2509.06836].

## 1. Position in the pruning landscape

Prior pruning methods are presented as falling into two broad families. Depth pruning removes entire transformer layers; width pruning removes hidden dimensions such as FFN channels or attention heads. COMPACT is motivated by the limitations of both.

| Method family | Advantages | Limitations |
|---|---|---|
| Depth pruning | Retains standard architecture | Extremely coarse-grained; accuracy drops abruptly once critical layers are removed |
| Width pruning | Fine-grained; smooth degradation | Breaks the uniform layer layout and thus requires custom inference kernels and code |
| COMPACT | Keeps a standard transformer architecture; scale-adaptive; training-free | Hybrid depth+width schemes may still offer higher absolute speedups |

COMPACT is designed to deliver memory savings by reducing both the vocabulary and the FFN intermediate dimension, latency improvements by shrinking key matrix multiplies, deployment ease because pruned models keep the exact same transformer skeleton, scale adaptivity because small models and large models expose different parameter bottlenecks, and training-free operation because it requires only a small calibration set and forward passes—no gradient computations or weight updates [2509.06836]. This suggests a deliberate attempt to retain the deployment advantages of depth pruning while recovering the smoother degradation profile associated with width pruning.

## 2. Formal setting and pruning targets

The method assumes a pre-trained decoder-only transformer with vocabulary size \(V\), embedding and unembedding matrices of size \(V \times D\), and \(L\) transformer blocks, each with hidden dimension \(D\) and FFN intermediate dimension \(I\) [2509.06836]. The pruning objective is to move to a smaller vocabulary size \(V' < V\) and a smaller intermediate dimension \(I' < I\), reducing parameter count and inference cost while preserving accuracy under the post-pruning token distribution.

A central design feature is scale adaptivity. Small models, described as having a high \(V/D\) ratio, rely more on embedding pruning through larger reductions in \(V'\). Large models, described as having large \(L D I\), rely more on FFN pruning through larger reductions in \(I'\) [2509.06836]. The paper states the parameter-growth asymmetry explicitly: small models have proportionally more vocabulary parameters \(O(V D)\), whereas large models have more FFN parameters \(O(L D I)\). A plausible implication is that COMPACT treats vocabulary pruning and FFN pruning as complementary budget-allocation mechanisms rather than as independent compression steps.

## 3. Common-token pruning across vocabulary and channels

COMPACT begins from the observation that BPE token frequencies follow Zipf’s law, so that most tokens are extremely rare [2509.06836]. Let \(S \subset V\) denote the set of \(V - V'\) rarest tokens, ranked by corpus frequency \(p(w)\). The vocabulary-pruning stage removes the corresponding rows of the embedding matrix and the corresponding columns of the unembedding matrix. The stated rationale is that tokens in \(S\) will never be produced or consumed, so this is lossless for common-token accuracy. No calibration data or learning is required at this stage.

The FFN-pruning stage adapts the standard activation-based channel score to the post-pruning token distribution. The baseline “act\(^2\)” score for channel \(k\) is

$$
I_k = \sum_i \left\| \left(\sigma(X_i W_1) \odot (X_i W_2)\right)_k \right\|^2 .
$$

COMPACT modifies this by assigning zero weight to any example whose input token \(x_i \in S\). With

$$
w_i =
\begin{cases}
0 & \text{if } x_i \in S \\
1 & \text{otherwise},
\end{cases}
$$

the score becomes

$$
I_k = \sum_i w_i \cdot \left\| \left(\sigma(X_i W_1) \odot (X_i W_2)\right)_k \right\|^2 .
$$

This “common-act\(^2\)” focuses channel importance on the distribution of tokens that remain valid post-vocab-pruning [2509.06836].

The joint optimization exposes two knobs for a target pruning ratio \(r\): \(V'\), which controls vocab pruning, and \(I'\), which controls FFN channel pruning. Hyperparameter sweeps then identify the pair \((V', I')\) that meets the target ratio while maximizing downstream accuracy. The pruning procedure is specified as follows: determine \(S\) as the \(V - V'\) rarest tokens; run forward passes on a calibration dataset and accumulate the channel scores; for each layer, prune the \(I - I'\) channels with lowest scores by removing corresponding rows in \(W^\ell_1\) and \(W^\ell_2\) and columns in the downstream projection; prune the embeddings and unembeddings for tokens in \(S\); and return the pruned model [2509.06836].

## 4. Architecture preservation and implementation characteristics

A defining implementation claim is architecture preservation. COMPACT removes only entire rows and columns, leaving layer counts, attention shapes, residual connections, and layer-norm layers untouched [2509.06836]. The method is also described as keeping the same number of layers and the same shape for attention, query, and projection matrices, so the model retains the exact same transformer skeleton. This property is presented as the basis for deployment-friendliness.

Because the pruned matrices still conform to standard dense GEMMs and embedding lookups, the method is reported to work out-of-the-box with standard transformer frameworks including Huggingface, vLLM, Triton, and TensorRT, without specialized kernels or custom inference code [2509.06836]. This directly addresses a common practical concern about width pruning, namely that fine-grained structural changes often require nonstandard kernels or framework modifications.

The method is training-free in the strict sense used in the paper: no back-propagation or retraining is performed, and only forward inference and channel scoring are required [2509.06836]. For 8B–70B models, pruning runs are reported as approximately \(30\,\mathrm{s}\)–\(2\,\mathrm{min}\) on a single GPU, and the total wall-clock time is also reported as approximately \(0.5\)–\(2\,\mathrm{minutes}\) on one A100 GPU. The runtime complexity of pruning is stated as \(O(|D|^2 L + |D| V)\) for calibration forward passes; scoring is linear in the number of channels times calibration samples.

## 5. Empirical performance and operating regime

The evaluation covers Qwen 2.5 (0.5B), LLaMA 3.2 (1B), Gemma 3 (1B), and LLaMA 3.1 (8B, 70B), with downstream benchmarks including MMLU, HellaSwag, WinoGrande, ARC-C/E, PIQA, and GSM8K [2509.06836]. Results are reported at pruning ratios of approximately 10%, 20%, and 35%.

At these pruning ratios, COMPACT is reported to maintain average downstream accuracy at \(\ge 90\%\) of dense at 10%, \(\ge 80\%\) at 20%, and \(\ge 63\%\) at 35%, across all model sizes [2509.06836]. Baselines such as ShortGPT, LaCo, SliceGPT, and 2SSP are described as collapsing around 20–35% on small models, while depth-pruned methods exhibit step-like accuracy drops. On small LLMs from 0.5B to 1B, COMPACT delays accuracy collapse beyond 35% pruning, whereas SliceGPT and 2SSP collapse at 10–20%. On large LLMs from 8B to 70B, COMPACT slightly outperforms 2SSP and depth-pruned methods in average accuracy, with smoother degradation curves.

The paper also reports concrete system-level gains on LLaMA 3.1-8B at 35% pruning:

| Metric | Dense | COMPACT |
|---|---:|---:|
| GPU memory usage | 50 GB | 32 GB |
| Throughput (classification) | 147 q/s | 201 q/s |
| Throughput (generation) | 81 q/s | 112 q/s |

These figures correspond to GPU memory reduced to \(0.64\times\), classification throughput of \(1.37\times\), and generation throughput of \(1.38\times\) relative to the dense model [2509.06836]. At larger scales, the paper summarizes the effect as \(0.65\)–\(0.75\times\) memory and approximately \(1.3\)–\(1.5\times\) speedups compared to dense. Pruning time on 70B is reported as 2 minutes, which is stated to be \(6\times\) faster than 2SSP and comparable to depth methods such as LaCo and ShortGPT.

## 6. Limitations, misconceptions, and prospective extensions

The paper’s conclusion emphasizes three limitations and follow-on directions [2509.06836]. First, while width pruning yields throughput improvements over the dense baseline, depth pruning still yields higher absolute speedups because removing entire layers gives greater parallelism gains. Hybrid schemes that combine partial depth pruning with COMPACT width pruning are therefore presented as a possible way to close this gap.

Second, calibration-data sensitivity is described as low, with 16–256 samples sufficing, but the automated tuning of \((V', I')\) remains heuristic. The text suggests meta-learning or analytical budget allocation as future mechanisms for streamlining hyperparameter selection. This indicates that scale adaptivity is already present at the method level, but its budget-allocation policy is not yet fully automated.

Third, recovery fine-tuning is optional rather than intrinsic. Optional self-distillation (SDD) can recoup an extra approximately 13% average performance at 35% pruning, but this incurs additional compute and dataset creation [2509.06836]. The paper further suggests exploring lightweight adapters or quantization in conjunction with COMPACT.

A frequent misconception in the pruning literature is that width pruning necessarily breaks standard deployment stacks. COMPACT is explicitly presented as a counterexample: it prunes vocabulary and FFN channels while preserving the standard transformer layout, standard dense GEMMs, and compatibility with existing inference frameworks [2509.06836]. Another misconception is that training-free pruning is necessarily too coarse to preserve accuracy at moderate pruning ratios. The reported results at approximately 10%, 20%, and 35% pruning suggest otherwise, although the method still inherits the broader trade-off that higher compression eventually reduces downstream accuracy.

In sum, COMPACT defines a joint pruning framework in which vocabulary reduction and FFN channel pruning are coordinated around the common-token distribution. Its main contribution is not merely parameter removal, but a pruning criterion that reweights channel importance after rare-token excision while preserving a standard transformer skeleton, enabling substantial memory savings and throughput gains without retraining or custom inference infrastructure [2509.06836].

Source: https://www.emergentmind.com/topics/compact-fe023ef7-94db-40a5-b5ed-ddfb006e804e