Static Width Pruning

• Static width pruning is a structured method that reduces neural network width by selecting key channels, dimensions, or units for a fixed deployment configuration.
• It employs techniques like learnable scaling factors, numerical channel optimization, and PCA-based transforms to prune filters, heads, and embedding dimensions across various models.
• The approach improves realized latency and storage efficiency, though its effectiveness varies with task demands and hardware constraints.

Static width pruning is a family of structured compression methods that reduces representational width before deployment and then fixes the result for all subsequent inference or retrieval. In convolutional networks, the pruned unit is typically the filter or channel; in transformers it may be a projection dimension, attention head, MLP hidden dimension, or KV channel; in retrieval systems it may be an embedding dimension, an active term within a document representation, or a posting retained for a term. The common property is that the final artifact is static: after pruning, deployment uses a conventional narrower network, a reduced dense vector space, or a pruned index, rather than input-dependent gating or unstructured sparsity masks (Haider et al., 2020, Zhu et al., 2021, Siciliano et al., 2024).

1. Definition, scope, and formal distinction

A compact formalization appears in the reasoning-centric LLM study of static pruning, where width pruning keeps only a subset of hidden dimensions while preserving layer count,

$$\mathbf{H}_l' = \mathbf{H}_l[:, \mathcal{I}_l], \quad |\mathcal{I}_l| = d',$$

with $d' < d$ (Ding et al., 26 Jan 2026). In that formulation, width pruning is contrasted with depth pruning, which removes layers, and with dynamic pruning, in which the executed path changes with the input. The same distinction recurs across CNNs, ViTs, LLMs, retrieval encoders, and sparse indexes: training-time masks, gates, salience scores, or search procedures may be dynamic or stochastic, but the deployed result is a fixed-width structure (Haider et al., 2020, Pan et al., 2023).

The term “width” is therefore broader than uniform hidden-size reduction. In the cited literature it refers to filter/channel counts in CNNs, linear-projection dimensions in ViTs, head and FFN widths in generative transformers, KV-channel width in decoder-only LLMs, dense embedding dimensionality in retrieval, and even support size in sparse retrieval indexes through per-document term budgets or per-term posting budgets (Zhu et al., 2021, Lv et al., 2024, Siciliano et al., 2024, Lassance et al., 2023, Pehlivan et al., 2013).

Domain	Width unit	Static deployed artifact
CNNs	Filters, channels, layer-wise channel counts	Conventional CNN with fewer filters/channels
Transformers and LLMs	Projection dimensions, heads, MLP hidden dimensions, KV channels	Dense pruned model or fixed compressed student
Retrieval and indexing	Embedding dimensions, active terms, postings	Reduced vector index or statically pruned inverted index

A central conceptual distinction is that static width pruning targets structured removals that preserve implementation regularity after model surgery. This separates it from unstructured weight sparsity, and also from dynamic schemes whose per-input decisions can increase indexing overhead, memory-access cost, or routing complexity even when nominal FLOPs decrease (Haider et al., 2020, Pan et al., 2023).

2. Canonical CNN formulations

A canonical CNN instantiation is “Comprehensive Online Network Pruning via Learnable Scaling Factors” (Haider et al., 2020). Its width-pruning component attaches a learnable scaling factor to each convolutional filter output and multiplies the pruning tensor with the convolution output so that each channel $y_i$ becomes $\phi_i y_i$. The paper writes the dropout-inspired form as

$$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$

and optimizes the supervised objective plus a sparsity penalty,

$$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$

In the reported experiments, $\mathcal{L}_D$ is cross-entropy and the sparsity term is instantiated as $L_1$ regularization on the pruning tensors with weight $=1/$num of filters. Training is online, but deployment is static: after 100 epochs with pruning tensors present, filters whose learned gates fall below an empirically chosen threshold are physically removed, the pruning tensors are deleted, and the model is fine-tuned for 10 epochs (Haider et al., 2020).

The resulting pruned architecture is a standard CNN with fewer channels. On VGG-16, the paper reports 88.27% accuracy with 83.05% parameter reduction and 40.21% FLOPs reduction on CIFAR-10; 97.57% accuracy with 97.36% parameter reduction and 89.60% FLOPs reduction on MNIST; and 92.16% accuracy with 92.95% parameter reduction and 69.16% FLOPs reduction on Fashion-MNIST (Haider et al., 2020). It also observes that later VGG layers are pruned much more aggressively than earlier ones, interpreting early filters as reusable low-level features and later filters as more task-specific and redundant.

A distinct formulation appears in “Joint Multi-Dimension Pruning via Numerical Gradient Update” (Liu et al., 2020). There, width is parameterized directly as layer-wise channel counts inside a pruning vector

$\vec{v} = \{c,s,d\},$

or more explicitly $d' < d$0. Rather than learning channel gates, the method optimizes numerical channel numbers jointly with spatial resolution and depth, then instantiates the network by keeping the first $d' < d$1 channels and cropping weight tensors correspondingly. This yields a static-width network after retraining from scratch. On ResNet-50, the width-only setting reports 76.2% at 3.0G FLOPs, 75.6% at 2.0G FLOPs, and 73.4% at 1.0G FLOPs, while the joint width+spatial+depth setting improves those trade-offs further (Liu et al., 2020). This suggests that static width pruning can be framed either as gate learning over channels or as direct architecture optimization over layer-wise widths.

3. Dynamic training, static deployment, and the realism of acceleration

“Progressive Channel-Shrinking Network” makes the dynamic-to-static conversion explicit (Pan et al., 2023). During training, each layer produces a salience vector through a lightweight SE-like generator,

$d' < d$2

and the output channels are reweighted as $d' < d$3. Instead of hard thresholding, the method sorts salience values and progressively shrinks the smallest $d' < d$4 entries using

$d' < d$5

with a scheduled shrinking coefficient. The decisive step for static width pruning is the Running Shrinking Policy: a moving average of salience vectors,

$d' < d$6

is converted after training into a fixed Boolean mask, and weights are physically pruned so that the deployed tensor has reduced shape

$d' < d$7

The training process is hybrid; inference is static (Pan et al., 2023).

The practical importance of that static conversion is visible in the reported latency results. On ImageNet with ResNet-18, input-dependent shrinking and running shrinking obtain the same 29.8% error, but running shrinking increases MAdds slightly from 0.9G to 1.0G while reducing MAC from 15.1M to 6.9M and measured latency from 19.1 ms to 13.8 ms on GPU and from 37.5 ms to 26.6 ms on ARM (Pan et al., 2023). The paper’s argument is therefore not merely that width can be reduced, but that a testing-static channel configuration converts nominal complexity reduction into lower memory-access cost and better realized latency.

A broader systems benchmark reaches a related conclusion from a different angle. “Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy” reorganizes pruning by GEMM dimensions $d' < d$8, $d' < d$9, and $y_i$0, and treats the practically important width family as static $y_i$1 pruning, where output-feature pruning in one GEMM propagates as input-feature pruning in the next (Hu et al., 8 Jun 2026). The paper finds that static depth pruning remains the strongest Pareto-optimal baseline at low quality loss, while static width pruning becomes competitive only at higher loss levels: static $y_i$2 reaches 1.51x speedup at 17.27% loss and 1.77x at 26.41% loss. This directly rejects the common equation of FLOPs reduction with deployable speedup: the acceleration value of static width pruning depends strongly on how reduced widths interact with tiling, alignment, and operator coverage (Hu et al., 8 Jun 2026).

4. Transformer, LLM, and generative-model instantiations

In Vision Transformers, “Vision Transformer Pruning” applies static width pruning to linear-projection dimensions rather than to channels in a CNN (Zhu et al., 2021). It introduces learnable dimension-wise importance scores, adds an $y_i$3 term,

$y_i$4

then globally ranks scores, thresholds them at $y_i$5, and physically rebuilds reduced projections $y_i$6. The pruned DeiT-B remains a dense transformer, but widths can differ by layer and submodule rather than collapsing to one global hidden size. On ImageNet-1K, the paper reports that 40% pruning yields 48.0M parameters, 10.0B FLOPs, and 80.7% Top-1, compared with the 86.4M-parameter, 17.6B-FLOP, 81.8%-Top-1 baseline (Zhu et al., 2021).

In decoder-only LLMs, “KVPruner” narrows the attention width specifically to relieve KV-cache pressure (Lv et al., 2024). It allocates a block-wise pruning ratio using perplexity sensitivity,

$y_i$7

and scores channels by averaging their $y_i$8 importance,

$y_i$9

The result is one-shot static pruning with no additional runtime overhead. On LLaMA-7B, 50%(qkv) pruning reduces KV store from 8.0 GB to 4.0 GB and parameters from 7.0B to 5.5B, while increasing A100 throughput from 63.02 to 78.111 tokens/s; the reported WikiText2 perplexity becomes 21.50 relative to 12.61 for the dense baseline (Lv et al., 2024).

Width pruning has also been specialized to generative transformers for video. In PARE, width refers to self-attention heads, cross-attention heads, and FFN hidden dimensions inside each block of Wan2.1-14B; the student keeps all 40 blocks but narrows those sublayers (Wang et al., 26 May 2026). Self-attention heads are scored with a calibration-based WANDA-style criterion and then corrected by temporal-head protection,

$\phi_i y_i$0

with $\phi_i y_i$1; FFN neurons are scored structurally by

$\phi_i y_i$2

and filtered for diversity. After physical submatrix extraction and width distillation, the width-only student achieves about 30% parameter reduction. On text-to-video, the width-only model reports VBench Avg 77.46 versus 77.70 for the teacher; on image-to-video, width-only reports 77.09 versus 77.92 (Wang et al., 26 May 2026). This is a recent example of static width pruning as structure-aware intra-block compression rather than merely global hidden-size reduction.

5. Retrieval and index-level variants

Static width pruning also appears outside neural hidden states. In dense retrieval, “Static Pruning in Dense Retrieval using Matrix Decomposition” reduces embedding dimensionality from $\phi_i y_i$3 to $\phi_i y_i$4 with a query-independent PCA transform (Siciliano et al., 2024). With document embeddings arranged in $\phi_i y_i$5, the method computes

$\phi_i y_i$6

projects documents to $\phi_i y_i$7, and applies the same fixed transform to queries,

$\phi_i y_i$8

The reduced system remains static because the same subspace is used for every query. The paper reports over 50% dimensionality reduction with up to a 5% reduction in NDCG@10, and emphasizes that the expensive document-side work is fully offline (Siciliano et al., 2024).

In sparse neural retrieval, width corresponds to support size in the inverted index. “A Static Pruning Study on Sparse Neural Retrievers” evaluates document-centric, term-centric, and agnostic pruning for DeepImpact, uniCOIL, and SPLADE (Lassance et al., 2023). Document-centric pruning retains only the top $\phi_i y_i$9 terms per document; term-centric pruning shortens each postings list by removing low-impact postings within the term; agnostic pruning applies one global threshold. The paper reports that static pruning achieves $$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$0 speedup with negligible effectiveness loss ($$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$1 drop) and, depending on the use case, even $$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$2 speedup with minimal impact ($$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$3 drop). In this setting, document-centric pruning is the clearest analogue of width capping, because every document is truncated to a fixed maximum active-term budget (Lassance et al., 2023).

A more principled index-level formulation appears in diversification-based static pruning for temporal collections (Pehlivan et al., 2013). There the retained postings set $$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$4 of size $$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$5 is chosen by maximizing expected utility,

$$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$6

with expectation taken over queries and aspects, and with DCG as the target metric. The deployed form is still a per-term top-$$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$7 retained postings set, but which postings survive is determined by diversification rather than raw score alone. This suggests that, at the index level, static width pruning can be viewed either as simple support truncation or as support-budget optimization under a retrieval objective (Pehlivan et al., 2013).

6. Empirical regularities, task dependence, and persistent misconceptions

One recurring result is that width should not be studied in isolation from other compression axes. For iterative magnitude pruning, “On the Predictability of Pruning Across Scales” finds that width, depth, and density combine through an approximately error-preserving invariant,

$$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$8

At fixed depth, equal-error width–density tradeoffs satisfy $$\mathbf{h}_l = f(\mathbf{h}_{l-1}\odot g(\Phi_{l-1}^f,\epsilon)),$$9, so wider models can tolerate more aggressive pruning and narrower ones must generally remain denser (Rosenfeld et al., 2020). Although that study is about unstructured IMP rather than structured channel pruning, it supports a broader interpretation: width is one coordinate on a trade-off surface, not a standalone knob.

A second recurring result is that width pruning is strongly task-dependent. In the controlled LLM/LRM comparison, static depth pruning outperforms width pruning on classification, while width pruning is more robust for generation and reasoning; static pruning also preserves reasoning better than dynamic pruning (Ding et al., 26 Jan 2026). The paper attributes that asymmetry to path continuity: depth pruning shortens the computation chain, whereas width pruning preserves full depth and mainly reduces representational capacity. The same study reports that SliceGPT consistently outperforms LLM-Pruner across all pruning ratios on both LLM-instruct and LLM-think models (Ding et al., 26 Jan 2026).

A third result is that “uniform degradation” is not a reliable description of static width pruning. “Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2” prunes GLU-MLP width using the Maximum Absolute Weight criterion and reports a systematic split between capabilities (Martra, 27 Dec 2025). Performance on MMLU, GSM8K, and perplexity metrics degrades predictably as the expansion ratio is reduced, but instruction-following improves by +46% to +75% in IFEval for Llama-3.2-1B and 3B, MUSR remains robust, and TruthfulQA-MC2 improves. The paper further reports a strong inverse correlation in Llama-3B between MMLU and TruthfulQA-MC2, $$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$0 (Martra, 27 Dec 2025). This does not imply that width pruning is generally beneficial; it shows instead that width can act as a capability-selective intervention.

Several practical limitations recur across the literature. Thresholds are often chosen empirically after sparse training rather than derived from a constrained optimum (Haider et al., 2020). Calibration choice can materially affect outcomes, especially for subspace-based width reduction such as SliceGPT-style residual-stream compression (Ding et al., 26 Jan 2026). Realized latency depends on alignment and kernel behavior rather than on FLOPs alone, and unaligned layouts can erase much of the nominal acceleration (Hu et al., 8 Jun 2026). Moreover, not every method that reduces “weight width” is a pure static-width method in the narrow dense-architecture sense: the FPGA ViT co-design based on block-structured pruning in attention and column/row pruning in MLP is only partially a width-pruning method, because much of its execution model remains structured sparse rather than simply narrower dense tensors (Parikh et al., 2024).

A final conceptual extension comes from “Bayesian Bits,” which treats $$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$1-bit quantization as pruning through nested stochastic gates,

$$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$2

Here $$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$3 implies $$\mathcal{L}(W,\Phi) = \mathcal{L}_D(W,\Phi) + \lambda^f\mathcal{L}_s(\Phi^f).$$4, so pruning becomes one endpoint of a static resource-allocation hierarchy that also includes mixed precision (Baalen et al., 2020). This suggests a broader interpretation of static width pruning: not merely binary keep/drop selection, but part of a larger family of offline, hardware-aware structural capacity allocation methods.

Static width pruning is therefore best understood as a deployment-oriented principle rather than a single algorithm. Across CNNs, transformers, LLMs, video generators, dense retrievers, and sparse indexes, its defining property is the production of a fixed reduced-width structure. The main open questions concern how to choose those widths under realistic latency constraints, how to reconcile width with depth and routing decisions, and how to exploit the evident task dependence of width reduction without mistaking nominal sparsity for actual acceleration or assuming that all capabilities degrade uniformly.