COMPACT: Optimized Transformer Pruning

• COMPACT is a joint pruning method that removes rare vocabulary tokens and prunes FFN channels using common-token activations while preserving the standard transformer architecture.
• It is training-free and scale-adaptive, delivering significant memory savings and increased throughput with minimal loss in downstream accuracy.
• By pruning entire rows and columns, COMPACT maintains compatibility with standard transformer frameworks, ensuring easy deployment and efficient inference.

COMPACT, short for “Common-token Optimized Model Pruning Across Channels and Tokens,” is a pruning method for decoder-only transformers that jointly prunes rare vocabulary and feed-forward network (FFN) intermediate channels in order to reduce memory, latency, and serving cost while preserving a standard transformer architecture (Kwek et al., 8 Sep 2025). The method combines two operations: removing the rarest vocabulary entries from the embedding and unembedding matrices, and pruning FFN channels using common-token-weighted activations so that channel importance is aligned with the post-pruning token distribution. It is designed to be training-free, scale-adaptive, and deployment-friendly, and experiments across Qwen, LLaMA, and Gemma families from 0.5B to 70B report state-of-the-art downstream task performance at similar or higher pruning ratios together with substantial reductions in parameters, GPU memory, and end-to-end latency (Kwek et al., 8 Sep 2025).

1. Position in the pruning landscape

Prior pruning methods are presented as falling into two broad families. Depth pruning removes entire transformer layers; width pruning removes hidden dimensions such as FFN channels or attention heads. COMPACT is motivated by the limitations of both.

Method family	Advantages	Limitations
Depth pruning	Retains standard architecture	Extremely coarse-grained; accuracy drops abruptly once critical layers are removed
Width pruning	Fine-grained; smooth degradation	Breaks the uniform layer layout and thus requires custom inference kernels and code
COMPACT	Keeps a standard transformer architecture; scale-adaptive; training-free	Hybrid depth+width schemes may still offer higher absolute speedups

COMPACT is designed to deliver memory savings by reducing both the vocabulary and the FFN intermediate dimension, latency improvements by shrinking key matrix multiplies, deployment ease because pruned models keep the exact same transformer skeleton, scale adaptivity because small models and large models expose different parameter bottlenecks, and training-free operation because it requires only a small calibration set and forward passes—no gradient computations or weight updates (Kwek et al., 8 Sep 2025). This suggests a deliberate attempt to retain the deployment advantages of depth pruning while recovering the smoother degradation profile associated with width pruning.

2. Formal setting and pruning targets

The method assumes a pre-trained decoder-only transformer with vocabulary size $V$, embedding and unembedding matrices of size $V \times D$, and $L$ transformer blocks, each with hidden dimension $D$ and FFN intermediate dimension $I$ (Kwek et al., 8 Sep 2025). The pruning objective is to move to a smaller vocabulary size $V' < V$ and a smaller intermediate dimension $I' < I$, reducing parameter count and inference cost while preserving accuracy under the post-pruning token distribution.

A central design feature is scale adaptivity. Small models, described as having a high $V/D$ ratio, rely more on embedding pruning through larger reductions in $V'$. Large models, described as having large $L D I$, rely more on FFN pruning through larger reductions in $V \times D$0 (Kwek et al., 8 Sep 2025). The paper states the parameter-growth asymmetry explicitly: small models have proportionally more vocabulary parameters $V \times D$1, whereas large models have more FFN parameters $V \times D$2. A plausible implication is that COMPACT treats vocabulary pruning and FFN pruning as complementary budget-allocation mechanisms rather than as independent compression steps.

3. Common-token pruning across vocabulary and channels

COMPACT begins from the observation that BPE token frequencies follow Zipf’s law, so that most tokens are extremely rare (Kwek et al., 8 Sep 2025). Let $V \times D$3 denote the set of $V \times D$4 rarest tokens, ranked by corpus frequency $V \times D$5. The vocabulary-pruning stage removes the corresponding rows of the embedding matrix and the corresponding columns of the unembedding matrix. The stated rationale is that tokens in $V \times D$6 will never be produced or consumed, so this is lossless for common-token accuracy. No calibration data or learning is required at this stage.

The FFN-pruning stage adapts the standard activation-based channel score to the post-pruning token distribution. The baseline “act$V \times D$7” score for channel $V \times D$8 is

$V \times D$9

COMPACT modifies this by assigning zero weight to any example whose input token $L$0. With

$L$1

the score becomes

$L$2

This “common-act$L$3” focuses channel importance on the distribution of tokens that remain valid post-vocab-pruning (Kwek et al., 8 Sep 2025).

The joint optimization exposes two knobs for a target pruning ratio $L$4: $L$5, which controls vocab pruning, and $L$6, which controls FFN channel pruning. Hyperparameter sweeps then identify the pair $L$7 that meets the target ratio while maximizing downstream accuracy. The pruning procedure is specified as follows: determine $L$8 as the $L$9 rarest tokens; run forward passes on a calibration dataset and accumulate the channel scores; for each layer, prune the $D$0 channels with lowest scores by removing corresponding rows in $D$1 and $D$2 and columns in the downstream projection; prune the embeddings and unembeddings for tokens in $D$3; and return the pruned model (Kwek et al., 8 Sep 2025).

4. Architecture preservation and implementation characteristics

A defining implementation claim is architecture preservation. COMPACT removes only entire rows and columns, leaving layer counts, attention shapes, residual connections, and layer-norm layers untouched (Kwek et al., 8 Sep 2025). The method is also described as keeping the same number of layers and the same shape for attention, query, and projection matrices, so the model retains the exact same transformer skeleton. This property is presented as the basis for deployment-friendliness.

Because the pruned matrices still conform to standard dense GEMMs and embedding lookups, the method is reported to work out-of-the-box with standard transformer frameworks including Huggingface, vLLM, Triton, and TensorRT, without specialized kernels or custom inference code (Kwek et al., 8 Sep 2025). This directly addresses a common practical concern about width pruning, namely that fine-grained structural changes often require nonstandard kernels or framework modifications.

The method is training-free in the strict sense used in the paper: no back-propagation or retraining is performed, and only forward inference and channel scoring are required (Kwek et al., 8 Sep 2025). For 8B–70B models, pruning runs are reported as approximately $D$4–$D$5 on a single GPU, and the total wall-clock time is also reported as approximately $D$6–$D$7 on one A100 GPU. The runtime complexity of pruning is stated as $D$8 for calibration forward passes; scoring is linear in the number of channels times calibration samples.

5. Empirical performance and operating regime

The evaluation covers Qwen 2.5 (0.5B), LLaMA 3.2 (1B), Gemma 3 (1B), and LLaMA 3.1 (8B, 70B), with downstream benchmarks including MMLU, HellaSwag, WinoGrande, ARC-C/E, PIQA, and GSM8K (Kwek et al., 8 Sep 2025). Results are reported at pruning ratios of approximately 10%, 20%, and 35%.

At these pruning ratios, COMPACT is reported to maintain average downstream accuracy at $D$9 of dense at 10%, $I$0 at 20%, and $I$1 at 35%, across all model sizes (Kwek et al., 8 Sep 2025). Baselines such as ShortGPT, LaCo, SliceGPT, and 2SSP are described as collapsing around 20–35% on small models, while depth-pruned methods exhibit step-like accuracy drops. On small LLMs from 0.5B to 1B, COMPACT delays accuracy collapse beyond 35% pruning, whereas SliceGPT and 2SSP collapse at 10–20%. On large LLMs from 8B to 70B, COMPACT slightly outperforms 2SSP and depth-pruned methods in average accuracy, with smoother degradation curves.

The paper also reports concrete system-level gains on LLaMA 3.1-8B at 35% pruning:

Metric	Dense	COMPACT
GPU memory usage	50 GB	32 GB
Throughput (classification)	147 q/s	201 q/s
Throughput (generation)	81 q/s	112 q/s

These figures correspond to GPU memory reduced to $I$2, classification throughput of $I$3, and generation throughput of $I$4 relative to the dense model (Kwek et al., 8 Sep 2025). At larger scales, the paper summarizes the effect as $I$5–$I$6 memory and approximately $I$7–$I$8 speedups compared to dense. Pruning time on 70B is reported as 2 minutes, which is stated to be $I$9 faster than 2SSP and comparable to depth methods such as LaCo and ShortGPT.

6. Limitations, misconceptions, and prospective extensions

The paper’s conclusion emphasizes three limitations and follow-on directions (Kwek et al., 8 Sep 2025). First, while width pruning yields throughput improvements over the dense baseline, depth pruning still yields higher absolute speedups because removing entire layers gives greater parallelism gains. Hybrid schemes that combine partial depth pruning with COMPACT width pruning are therefore presented as a possible way to close this gap.

Second, calibration-data sensitivity is described as low, with 16–256 samples sufficing, but the automated tuning of $V' < V$0 remains heuristic. The text suggests meta-learning or analytical budget allocation as future mechanisms for streamlining hyperparameter selection. This indicates that scale adaptivity is already present at the method level, but its budget-allocation policy is not yet fully automated.

Third, recovery fine-tuning is optional rather than intrinsic. Optional self-distillation (SDD) can recoup an extra approximately 13% average performance at 35% pruning, but this incurs additional compute and dataset creation (Kwek et al., 8 Sep 2025). The paper further suggests exploring lightweight adapters or quantization in conjunction with COMPACT.

A frequent misconception in the pruning literature is that width pruning necessarily breaks standard deployment stacks. COMPACT is explicitly presented as a counterexample: it prunes vocabulary and FFN channels while preserving the standard transformer layout, standard dense GEMMs, and compatibility with existing inference frameworks (Kwek et al., 8 Sep 2025). Another misconception is that training-free pruning is necessarily too coarse to preserve accuracy at moderate pruning ratios. The reported results at approximately 10%, 20%, and 35% pruning suggest otherwise, although the method still inherits the broader trade-off that higher compression eventually reduces downstream accuracy.

In sum, COMPACT defines a joint pruning framework in which vocabulary reduction and FFN channel pruning are coordinated around the common-token distribution. Its main contribution is not merely parameter removal, but a pruning criterion that reweights channel importance after rare-token excision while preserving a standard transformer skeleton, enabling substantial memory savings and throughput gains without retraining or custom inference infrastructure (Kwek et al., 8 Sep 2025).