Structured Pruning and Folding

• Structured pruning and folding are compression techniques that reduce network size by eliminating redundant neurons, filters, or channels while preserving predictive performance.
• Structured pruning relies on saliency criteria to remove entire structural components, whereas folding clusters similar elements using low-rank projections to form a compressed representation.
• Both methods enable hardware-efficient models with improved memory organization and are supported by theoretical guarantees linking projection error to functional performance.

Structured pruning and folding are compression methodologies used to reduce the computational and memory footprint of neural networks by eliminating redundant or non-essential structural components—typically at the neuron, filter, channel, or block level—while aiming to preserve the original model’s predictive performance. Whereas unstructured pruning induces fine-grained sparsity with irregular nonzero patterns, structured pruning and folding operate on coarse-grained axes (neurons, filters, heads), producing compressed models that are more hardware-efficient and, in the case of folding, more aware of the underlying geometry of the parameter space. Recent work has formalized both approaches as orthogonal projections in parameter space, elucidating the trade-offs between axis-aligned and low-rank projections and providing theoretical and empirical support for their roles in deployment-scale model compression (Gonzalez-Carabarin et al., 2021, Hu et al., 16 Jan 2025, Saukh et al., 20 Feb 2026).

1. Methodological Foundations

Structured pruning eliminates entire channels, neurons, kernels, or blocks, enforcing regular sparsity patterns. In its archetypal form, pruning selects a subset of locations (e.g., rows of a weight matrix) based on a saliency criterion (often magnitude-based), zeroing out the remainder. This operation is equivalent to projecting the parameter matrix onto an axis-aligned coordinate subspace. Formally, for layer weights $W \in \mathbb{R}^{m \times p}$, selecting $k$ channels corresponds to the orthogonal projection $C_p = \mathrm{diag}(1_k, 0_{m-k})$ and pruned weights $W_p = C_p W$, leaving only the $k$ chosen rows (Saukh et al., 20 Feb 2026).

Folding, in contrast, groups similar structural components (e.g., channels, neurons) using unsupervised clustering (typically $k$-means), replacing all members of a cluster with their centroid. This low-rank operation projects $W$ onto a subspace where only $k$ unique outputs remain, yielding $W_f = C_f W$ with $C_f = U_f(U_f^\top U_f)^{-1}U_f^\top$, where $k$0 is an indicator matrix for cluster assignments (Saukh et al., 20 Feb 2026). Folding is therefore a geometry-aware alternative that produces a compressed layer with only $k$1 unique rows.

Dynamic approaches such as Dynamic Probabilistic Pruning (DPP) generalize structured pruning to multiple granularities by learning differentiable, stochastic binary masks with exactly $k$2 nonzeros along a chosen pruning axis, using Gumbel-softmax relaxation for end-to-end training and mask sampling at weights, kernels, or feature-map level (Gonzalez-Carabarin et al., 2021). This supports highly regular memory layouts advantageous for hardware execution.

2. Projection Geometry and Theoretical Guarantees

Both structured pruning and folding can be interpreted as orthogonal projections in weight space. Pruning restricts the parameter matrix to a lower-dimensional axis-aligned coordinate subspace, while folding projects onto a lower-dimensional, typically non-axis-aligned, cluster-structured subspace. Theoretical results establish that, given any structured pruning of rank $k$3, there exists a folding to rank $k$4 with strictly smaller parameter reconstruction error $k$5 (Saukh et al., 20 Feb 2026). For optimal $k$6-means folding, this bound is tight: folding yields minimum distortion of the parameter matrix at the same or nearly the same rank as pruning.

Moreover, if the downstream loss $k$7 is $k$8-Lipschitz in Frobenius norm, smaller parameter projection error directly implies a tighter bound on functional perturbation—hence, folding is theoretically guaranteed to produce less worst-case degradation in loss than pruning, up to one-rank slack (Saukh et al., 20 Feb 2026).

3. Algorithmic Implementations

Structured Pruning

• Saliency Metrics: Structured pruning typically relies on $k$9- or $C_p = \mathrm{diag}(1_k, 0_{m-k})$0-norm saliency scores computed per neuron, channel, or filter. The $C_p = \mathrm{diag}(1_k, 0_{m-k})$1 units with highest scores are retained; others are zeroed and subsequently removed (Hu et al., 16 Jan 2025, Saukh et al., 20 Feb 2026).
• Inter-Layer Folding in Pruning: In models with sequential linear layers (e.g., MLPs, Transformer FFNs), removing a column in one layer allows simultaneous removal of the corresponding row in the preceding layer—a process sometimes called “folding out” the dimension. This preserves functional equivalence as the removed neuron’s contribution is exactly zero (Hu et al., 16 Jan 2025).
• Dynamic Probabilistic Pruning: DPP learns mask logits at each layer, sampling exactly $C_p = \mathrm{diag}(1_k, 0_{m-k})$2 of $C_p = \mathrm{diag}(1_k, 0_{m-k})$3 candidates (at arbitrary granularity) using Gumbel-Softmax relaxation. The mask is multiplied elementwise with the weight tensor, yielding pruned weights. DPP enables hardware-aligned sparsity by enforcing regular $C_p = \mathrm{diag}(1_k, 0_{m-k})$4-out-of-$C_p = \mathrm{diag}(1_k, 0_{m-k})$5 nonzero block patterns, with efficient memory organization and potential for joint quantization (Gonzalez-Carabarin et al., 2021).

Folding

• Model Folding via Clustering: Each output parameter (row of $C_p = \mathrm{diag}(1_k, 0_{m-k})$6) is embedded as a $C_p = \mathrm{diag}(1_k, 0_{m-k})$7-dimensional vector. $C_p = \mathrm{diag}(1_k, 0_{m-k})$8-means is applied to these rows, producing clusters $C_p = \mathrm{diag}(1_k, 0_{m-k})$9. All rows within a cluster are replaced by their mean; this operation is projected via the previously described $W_p = C_p W$0 matrix. The layer now implements a function dependent only on the $W_p = C_p W$1 unique outputs, which can be propagated as a structural reduction into subsequent layers (Saukh et al., 20 Feb 2026).
• Computational Efficiency: $W_p = C_p W$2-means folding is computationally feasible, typically running in time $W_p = C_p W$3 for $W_p = C_p W$4 epochs, with overhead negligible relative to training (Saukh et al., 20 Feb 2026).

Fast Structured Pruning and Weight Folding for LLMs

FASP (“Fast and Accurate Structured Pruning”) unites these ideas: it prunes columns in a weight matrix and “folds out” corresponding structural units—removing the same index row in the immediately preceding matrix—resulting in a dimension-reduced, functionally equivalent sub-network. This is followed by a least-squares restoration step over calibration data for accuracy recovery, avoiding retraining (Hu et al., 16 Jan 2025).

4. Empirical Evaluation and Benchmarking

Empirical studies show distinct regimes of advantage for pruning versus folding. Folding typically yields higher post-compression accuracy, especially at moderate-to-high compression (e.g., 50–80% channel reduction). Quantitatively, compression experiments on ResNet18, ViT, and LLaMA demonstrate:

• At high sparsity, folding outperforms $W_p = C_p W$5 (MAG1) and $W_p = C_p W$6 (MAG2) norm-based structured pruning by 5–10 percentage points in accuracy (Saukh et al., 20 Feb 2026).
• In LLaMA, at 20% sparsity, folding reduces perplexity from ~54 to ~47, while pruning (MAG2) achieves only ~50; at 50% sparsity, folding yields larger improvements (Saukh et al., 20 Feb 2026).
• FASP achieves LLM pruning times of 3 minutes for LLaMA-7B and 15 minutes for LLaMA-30B on a single RTX 4090, outperforming SliceGPT and NASLLM in both speed and perplexity at equivalent sparsity (Hu et al., 16 Jan 2025).

The advantage of folding is most pronounced under moderate learning rates, sharpness-aware minimization, and flatter minima. For very low compression (<10%), both methods are nearly equivalent. Folding remains robust after quick fine-tuning or norm recalibration (Saukh et al., 20 Feb 2026).

5. Hardware Implications and Memory Organization

Structured pruning and folding facilitate efficient hardware deployment:

• Structured sparsity patterns map directly to burst reads and coalesced memory accesses, in contrast to unstructured approaches that require irregular address arithmetic and control logic (Gonzalez-Carabarin et al., 2021).
• DPP's $W_p = C_p W$7-out-of-$W_p = C_p W$8 structure enables contiguous storage of nonzeros, eliminating the need for zero-padding (as in ELLPACK) or coordinate dictionaries (as in COO). For coarse granularity, no additional indices are needed beyond a block header—enabling minimal memory overhead (Gonzalez-Carabarin et al., 2021).
• Folding, by fusing neuron outputs by cluster, reduces not only storage but also runtime compute due to collapsed dimensions in the subsequent linear projections (Saukh et al., 20 Feb 2026).

6. Practical Recommendations, Limitations, and Extensions

Folding is recommended for scenarios targeting moderate–high compression, calibration-free settings, and where a small clustering overhead is acceptable. Structured pruning remains preferable for extremely low compression or where axis-aligned removals are strictly required by hardware constraints (Saukh et al., 20 Feb 2026). In LLM compression pipelines using activation-aware metrics (e.g., Wanda, SparseGPT), structured pruning with dynamic restoration (as in FASP) remains state-of-the-art due to optimal trade-offs between tractability, simplicity, and accuracy preservation (Hu et al., 16 Jan 2025).

Extensions include combining folding with low-rank factorization, hierarchical folding (e.g., heads, blocks), and, in dynamic frameworks, further tiling optimizations of the regular nonzero patterns (Gonzalez-Carabarin et al., 2021, Hu et al., 16 Jan 2025).

7. Quantitative Comparison of Approaches

Methodology	Theoretical Projection	Computational Complexity	Best Use Regime
Structured Pruning (MAG1/MAG2)	Axis-aligned (coordinate)	$W_p = C_p W$9	Very low compression, HW-limited cases
Folding (k-means)	Cluster-structured (low rank)	$k$0	Moderate-high compression, calibration-free
Dynamic Probabilistic Pruning (DPP)	Flexible $k$1-out-of-$k$2 blockwise	$k$3(training); efficient inference	Multigranularity, HW-optimized, joint quantization
FASP (Weight Folding + Restoration)	Structured + inter-layer folding	$k$4 + least-squares	Fast, accurate LLM pruning

Folding is generally empirically and theoretically superior in high compression and flat-minima regimes. Structured pruning remains relevant where index overwrites are not feasible, activation-based saliency is essential, or toolchains limit to channel removal. Dynamic structured methods such as DPP expand the granularity spectrum and enable differentiable hardware-aligned compression. Empirical evidence and projection-theoretic principles position folding as a preferred method in calibration-free, geometry-rich compression tasks, with structured pruning as a robust, hardware-enabling baseline (Gonzalez-Carabarin et al., 2021, Hu et al., 16 Jan 2025, Saukh et al., 20 Feb 2026).