Papers
Topics
Authors
Recent
Search
2000 character limit reached

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Published 20 Feb 2026 in cs.LG and cs.AI | (2602.18116v1)

Abstract: Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Summary

  • The paper establishes a unified framework by interpreting both pruning and folding as orthogonal projections, showing that folding yields lower reconstruction error.
  • It demonstrates through extensive experiments on models like ResNet, ViT, and LLaMA that folding achieves higher post-compression accuracy and faster recovery during fine-tuning.
  • The analysis links folding’s advantage to lower loss landscape sharpness, confirming its robustness under various training settings and hyperparameters.

Projection Geometry and Model Compression: An Expert Analysis of "Cut Less, Fold More" (2602.18116)

Reformulating Structured Pruning and Folding as Orthogonal Projections

The paper delivers a rigorous framework for neural network compression, reconciling structured pruning and model folding through projection geometry. The authors formalize channel/filter pruning as an axis-aligned orthogonal projection, mapping parameter matrices onto coordinate-aligned subspaces. In contrast, folding is conceptualized as projection onto cluster-structured subspaces, achieved by parameter clustering (e.g. via kk-means), resulting in merged representations. Both paradigms are encapsulated in the projection matrix C\mathbf{C}, such that the compressed weights Wc=CW\mathbf{W}_c = \mathbf{C}\mathbf{W} minimize the Euclidean (Frobenius) deviation from the original weights.

Strong Theoretical Dominance of Folding

The central theoretical result is that, for any pruning scheme of rank kpk_p (retaining kpk_p filters/channels), there exists a folded model with rank kf=kp+1k_f = k_p + 1 whose parameter reconstruction error is strictly lower (in Frobenius norm). The practical delta in network size is negligible (relative increase of 1/kp1/k_p). Beyond that, folding via optimal kk-means further guarantees minimal distortion among all cluster-based projections. This formalism is backed by a tight functional perturbation bound; provided Lipschitz continuity of the loss, lower parameter error translates directly into a bound on the loss difference.

Empirically, loss and accuracy variation between pruning and folding is substantially larger than between adjacent ranks, confirming that folding's advantage is structural, not merely due to increased rank.

Empirical Evaluation: Folding's Edge Across Architectures and Hyperparameters

The study presents a comprehensive calibration-free evaluation over >1,000>1,000 checkpoints of ResNet18, PreActResNet18, ViT-B/32, CLIP ViT-B/32, and LLaMA-60M/130M (C4), spanning numerous training regimes (optimizers, learning rates, augmentation, regularization, SAM). Compression is applied post-training, without access to training data or gradients.

Folding consistently achieves higher post-compression accuracy than magnitude pruning, especially at moderate–high compression ratios. Where performance gaps exist (e.g., very low compression or in highly augmented/SAM-flattened settings), the narrowing is attributed to increasingly invariant weight landscapes. Extensive ablations show the robustness of folding across optimizers (Adam vs SGD), learning rate schedules, regularization, and data augmentation.

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1

Figure 1: Folding yields superior post-compression accuracy relative to pruning, with the largest gains at moderate–high compression.

Retained Accuracy Advantage After Fine-tuning

Remarkably, folding not only preserves but enhances its lead after lightweight (LayerNorm/BatchNorm) and full fine-tuning. Folded models recover accuracy more rapidly and reach higher final performance post fine-tuning, indicating a better initialization and decreased perturbation under trainable adaptation.

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2: Folded models maintain a positive accuracy delta and exhibit faster recovery during fine-tuning.

Impact of Upstream Training Choices

Systematic ablation studies dissect how training hyperparameters modulate folding's advantage:

  • Learning Rate: Folding's performance peaks at moderate–low learning rates; at very high rates, pruning sometimes closes the gap.
  • Optimizer: Adam-trained models exhibit sharper minima and a more pronounced folding advantage; SGD-trained models are flatter and the gap decreases.
  • SAM: Sharpness-aware minimization reduces sharpness variability and narrows the accuracy gap; under large ρ\rho, differences between compression methods diminish.
  • Data Augmentation: Standard and RandAugment practices generally boost post-compression accuracy for both methods, though they narrow folding's advantage in ResNets while increasing it in ViTs.

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3: Learning rate modulates folding's edge; highest gains observed at moderate–low rates.

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4: Data augmentation tends to narrow folding's advantage in ResNets and amplify it in ViTs.

LLM Compression: Folding Outperforms Pruning in Small LLaMA Models

Compression results for LLaMA-60M and LLaMA-130M models on C4 indicate that folding delivers lower perplexity than magnitude pruning for FFN block compression, except in settings with very low learning rates and prolonged warmup. This positions folding as an effective, geometry-aware operator for transformer compression.

Sharpness and Loss Landscape Analysis

Sharpness measurements confirm that folding induces lower perturbation in the loss landscape, aligning with the theoretical Lipschitz bound. The accuracy difference correlates strongly with sharpness reduction under folding, particularly for Adam-trained models at moderate compression; for SGD, the correlation is weaker but folding still produces a gentler landscape trajectory.

Practical Implications and Future Directions

The projection-geometric perspective clarifies folding as a fundamentally richer, optimal operator class for model compression. Unlike pruning, which is restricted to axis-aligned coordinate removal, folding permits coordinated merging of parameters, minimizing distortion while maintaining FLOPs and memory equivalence at inference.

This framework opens new avenues for geometry-aware, calibration-free compression, and future work could include:

  • Extending folding to attention blocks in ViTs and LLMs.
  • Integrating folding into quantization and calibration-based pipelines.
  • Scaling empirical studies to larger LLMs and vision-LLMs (VLMs).
  • Designing hybrid schemes that combine folding, pruning, and distillation, optimizing for functional closeness.

Conclusion

"Cut Less, Fold More" (2602.18116) establishes folding as a superior, theoretically principled, and empirically validated alternative to structured pruning for calibration-free model compression. By interpreting both methods as orthogonal projections and proving folding's optimality on parameter and functional preservation, this work provides a foundation for advancing compression protocols applicable to modern architectures at scale. Empirical results consolidate folding's advantage across tasks, architectures, and hyperparameters, with detailed studies elucidating the connection between projection geometry, sharpness, and functional robustness. The projection-based formulation is extensible, suggesting potential for future developments in universal, geometry-aware compression for deep learning.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Cut Less, Fold More: A simple guide to the paper

What this paper is about (overview)

This paper looks at ways to make big neural networks smaller so they run faster and fit on devices with limited memory, like phones. The authors compare two popular ways to shrink models without using the training data again:

  • Pruning: cutting out parts of the model.
  • Folding: merging similar parts together.

Their main message: instead of “cutting” weights away (pruning), it’s often better to “fold” similar weights together (folding). They explain why this works using geometry and test it on many models. Most of the time, folding keeps accuracy higher, especially when you compress a lot.

What questions the paper asks (goals)

In simple terms, the paper asks:

  • If you want to shrink a model without retraining, is folding better than pruning?
  • Can we explain pruning and folding as two kinds of the same idea (simple geometry), so we can compare them fairly?
  • When does folding help most, and when might pruning do just as well?

How they studied it (methods, with analogies)

The authors use both math and experiments.

  1. A geometry view of compression Think of a model’s weights as points in space. Shrinking the model means pushing those points onto a smaller shape (a “subspace”), kind of like:
  • Projection analogy: shining a flashlight on a 3D object to get a 2D shadow. The “shadow” uses fewer dimensions but tries to stay close to the original.
  • Pruning as “axis-aligned projection”: like dropping the shadow straight down onto the floor by deleting some coordinates. This is simple but can throw away a lot of useful direction information.
  • Folding as “clustering projection”: first group similar weight vectors, then replace each group by their average. This keeps the groups’ directions and tends to stay closer to the original.
  1. A simple math promise (no heavy formulas) They prove that if you compare pruning that keeps k parts to folding that keeps roughly k+1 groups, folding will always stay at least as close—or closer—to the original model’s weights. Closer weights usually mean the model’s behavior changes less. This is like saying: if you allow folding one more “fold” than the number of parts pruning keeps, folding is guaranteed to distort the model less.
  2. Experiments across many models and settings They tested on more than 1,000 trained checkpoints, including:
  • Image classifiers: ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K.
  • LLMs: smaller LLaMA-family models (60M and 130M parameters) trained on the C4 dataset.

They also tried many training setups—different optimizers (Adam, SGD), learning rates, data augmentation, regularization, and a technique called SAM (Sharpness-Aware Minimization).

  1. Light touch after compression They mostly avoid retraining to keep the comparison fair (“calibration-free”), but they also test small fixes:
  • For CNNs: a quick BatchNorm recalibration (think of re-tuning the dials).
  • For ViTs: a quick LayerNorm reset.
  • A few experiments with short fine-tuning (1–5 epochs).

What they found and why it matters (results)

Main findings:

  • Folding often beats pruning: Across many models and settings, folding usually keeps accuracy higher after compression—especially at moderate to high compression levels.
  • Theory matches practice: The geometry idea explains the results. Folding keeps more of the original directions (by averaging similar weights instead of zeroing them), so the model changes less.
  • After small adjustments or brief fine-tuning, folding still holds a lead: Even when they let both methods do a small amount of clean-up or short fine-tuning, folding keeps its advantage and often recovers accuracy faster.
  • Not always one-sided: In a few special training setups—like very high learning rates, certain augmentations, or very low compression—the gap narrows and can sometimes flip.
  • LLMs too: On LLaMA-60M and 130M, folding often results in lower perplexity (meaning the model is less “confused” when predicting text) than pruning at the same compression levels.

Why this matters:

  • Better accuracy at the same size: If you need to deploy a model on a small device, folding helps you keep quality without extra data.
  • General and simple: Folding works in a “calibration-free” way, meaning you don’t need access to the training data, and it’s grounded in a clear geometric idea.

What this could change (impact)

  • For engineers: Folding can be a go-to compression step when retraining isn’t possible, helping keep performance high on edge devices.
  • For researchers: Viewing compression as “projections” opens the door to new methods that shape the compressed model to stay as functionally close as possible to the original.
  • For future work: The authors plan to apply folding to attention parts of transformer models, combine it with quantization or distillation, and test on bigger models.

Key takeaways for a 14-year-old

  • Pruning is like cutting parts off a paper model; folding is like folding similar parts together. Both make it smaller, but folding usually keeps the original shape better.
  • The authors prove a simple guarantee: with almost the same size, folding changes the model less than pruning.
  • In many tests with image and LLMs, folding kept accuracy higher—especially when shrinking a lot.
  • Sometimes pruning catches up, but folding is often the safer bet when you want small models that still work well.
View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The following points summarize what remains missing, uncertain, or unexplored in the paper and outline concrete directions for future research:

  • Matched-capacity guarantees: The theory establishes folding’s advantage with a one-rank slack (kf=kp+1k_f = k_p + 1), but provides no conditions for dominance at exactly matched parameter/FLOP budgets. Characterize necessary and sufficient conditions (e.g., on weight alignment, curvature, or layer structure) under which folding provably outperforms pruning at equal retained rank, or construct counterexamples.
  • Functional perturbation bounds beyond parameter-Lipschitz: Results hinge on a parameter-space Lipschitz assumption without quantifying κ\kappa or verifying local validity. Develop tighter, data-dependent function-space bounds (e.g., via NTK, Jacobian/Hessian spectra, local Lipschitz estimates) that connect Frobenius reconstruction error to task metrics (accuracy, perplexity) and hold under common activations (GELU, SiLU) and normalization.
  • Global compression allocation: Experiments primarily use uniform per-layer budgets. Formulate and solve the global budget allocation across layers (pruning vs. folding counts per-layer) to maximize accuracy under a total parameter/FLOP constraint; assess whether folding’s advantage persists under optimal cross-layer allocations.
  • Attention-layer compression: ViTs and LLMs are compressed only in FFN blocks. Design and evaluate folding for attention (Q/K/V projections, multi-head mixing, output projections), including how to merge heads and adapt subsequent layers, and compare against structured pruning of attention.
  • Post-folding adaptation mechanics: The paper assumes “join identical outputs while adapting the next layer” based on prior work. Provide a general, architecture-agnostic algorithm with correctness guarantees and complexity analysis for adapting downstream layers (including residual connections, skip paths, and normalization) after folding.
  • Interaction with advanced pruning methods: Comparisons are limited to magnitude-based structured pruning. Benchmark folding against calibration-based/second-order pruning (e.g., activation-aware, Hessian-aware), movement pruning, and layerwise sensitivity methods under matched budgets in both CNN/ViT and LLM settings.
  • Synergy with quantization and distillation: The paper does not study interactions with quantization or teacher–student distillation. Evaluate whether folding’s weight tying helps or hurts quantization codebook design, calibration accuracy, and distillation efficiency, and design joint optimization pipelines.
  • Unstructured sparsity and element-wise folding: Explore whether folding concepts extend to unstructured sparsity (e.g., clustering scalar weights akin to k-means quantization) and compare against fine-grained pruning/weight sharing in terms of accuracy and hardware efficiency.
  • Predictive metrics for method selection: Folding’s advantage varies with optimizer, learning rate, SAM radius, and augmentation. Develop predictive criteria (e.g., weight alignment indices, inter-channel cosine similarity, sharpness/curvature measures) to select between folding and pruning per model/layer before compression.
  • Stability and optimality of clustering: k-means initialization and non-convexity may affect folding outcomes. Quantify sensitivity to initialization, number of sweeps, and alternative clustering objectives (e.g., k-medoids, spectral clustering, cosine similarity) and assess their impact on reconstruction and accuracy.
  • Representation diversity and collapse: Folding ties weights within clusters and may reduce feature diversity. Measure intra-layer feature diversity, mutual information, and downstream transfer performance post-folding versus pruning; identify safeguards (e.g., diversity-aware clustering constraints).
  • Hardware and systems efficiency: Claims of equal inference compute at matched retained size are not validated on real hardware. Benchmark end-to-end latency, throughput, memory bandwidth, and kernel efficiency for folding versus structured pruning on GPUs/TPUs and sparsity accelerators; analyze deployability trade-offs.
  • Fine-tuning dynamics and budgets: Only short fine-tuning (1–5 epochs) is tested. Study longer fine-tuning schedules, learning rate warmups, and optimizer choices post-compression; characterize recovery dynamics and whether folding maintains its lead with extensive adaptation or under regularization constraints.
  • Larger-scale LLMs and downstream tasks: Results are limited to LLaMA 60M/130M on C4 perplexity. Extend to modern 7B–70B models and evaluate on downstream tasks (QA, summarization, coding pass@k, instruction following), including zero-shot/few-shot performance and safety alignment impacts.
  • Distribution shift and robustness: The paper does not assess out-of-distribution generalization, calibration, or adversarial robustness. Quantify how folding vs. pruning affects OOD accuracy, confidence calibration (ECE), adversarial robustness, and stability under input perturbations and data augmentations.
  • Non-ReLU activations and architectural breadth: Theory assumes ReLU and normalization layers. Generalize the projection framework and functional bounds to common activations (GELU, SiLU), gated units, normalization variants, and broader architectures (e.g., DenseNet, MoE, RNNs).
  • When pruning can be preferable: Empirically, there are corner cases where pruning narrows or reverses the gap. Theoretically and empirically characterize regimes (e.g., very high learning rates, strong augmentations, certain optimizers/datasets) in which pruning is better, and codify a decision procedure.
  • Layer-wise non-uniform compression strategies: Beyond uniform ratios, investigate learned or sensitivity-driven non-uniform allocations for folding (e.g., more clusters in critical layers) and compare to analogous pruning strategies.
  • Practical scalability of folding: While k-means overhead is modest at small scale, quantify compression-time cost and memory for very wide/deep networks, distributed settings, and mixed-precision training logs; explore scalable clustering (mini-batch, streaming, distributed) for billion-parameter models.
  • Formal link to low-rank decompositions: Folding is described as a low-rank projection via clustering, but relationships to SVD/Tucker/CP decompositions and oblique projections remain unclear. Establish theoretical connections and hybrid methods that combine clustering with matrix/tensor factorization.
  • Parameter-Lipschitz estimation: Provide practical procedures to estimate or bound the local parameter-Lipschitz constant κ\kappa per layer/model to make loss perturbation bounds actionable and comparable across training setups.
  • Multi-modal and VLM models: Extend evaluation to vision–LLMs (e.g., CLIP variants beyond ViT-B/32, LLaVA, Flamingo) and tasks (zero-shot retrieval, captioning), and assess folding’s impact on cross-modal alignment and performance.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following items translate the paper’s findings into concrete, deployable use cases. Each entry names the target sector(s), outlines a minimal workflow or product idea, and lists assumptions and dependencies that impact feasibility.

  • Edge and mobile vision models with better accuracy at moderate–high compression
    • Sectors: software, retail, AR/VR, robotics
    • What to do: Replace structured magnitude pruning with folding for CNNs/ViTs (e.g., ResNet18, ViT-B/32). Apply folding layer-wise (clustering rows/channels via k-means), adapt the next layer, then run a lightweight calibration step (REPAIR for BN or LayerNorm reset for ViTs). Optionally fine-tune for 1–5 epochs.
    • Tools/workflows: PyTorch/ONNX graph pass that clusters weights per layer; integrate folding pass into existing compression pipelines (e.g., TVM/TensorRT build step); automatic per-layer cluster counts to match parameter/FLOP budgets; “Fold+REPAIR” recipe for CNNs and “Fold+LN-reset” for ViTs.
    • Assumptions/dependencies: Most gains are at moderate–high compression; results shown for CNNs and ViT FFN blocks. Theoretical bounds rely on mild smoothness and normalization layers. Corner cases (very low compression or specific training regimes) can narrow the gap.
  • On-device small LLMs for assistants, summarizers, and chat
    • Sectors: software, education, customer support
    • What to do: Fold FFN blocks of 60M–130M parameter LLaMA-family models (20–50% layer-wise compression), then optionally brief fine-tuning. Use for mobile summarization, note-taking, or chat agents where memory and latency are constrained.
    • Tools/workflows: Folding pass for Transformer FFNs; automated budget selection per block; integration into mobile inference runtimes.
    • Assumptions/dependencies: Experiments target FFN blocks (not attention heads); larger LLMs and attention folding are not yet validated. Performance depends on training hyperparameters (e.g., learning rate and warmup).
  • Energy and cost savings in inference at scale
    • Sectors: cloud/edge platforms, energy, telecom
    • What to do: Use folding to maintain higher post-compression accuracy at a given FLOP/parameter budget, enabling smaller instances or higher throughput per node.
    • Tools/workflows: MLOps job that tests pruning vs folding at matched budgets; standardized dashboard tracking accuracy, Frobenius distance to original weights, and energy/latency.
    • Assumptions/dependencies: Folding’s compression step adds a one-time k-means clustering cost per layer (small compared to training). Inference compute is on par with pruning for matched retained sizes.
  • Privacy-preserving, data-free compression for regulated domains
    • Sectors: healthcare, finance, public sector
    • What to do: Adopt calibration-free folding (no training data required) to compress models where access to raw data is restricted. Use BN/LN reset when permissible; otherwise deploy as-is.
    • Tools/workflows: “No-data” compression pipeline with folding and optional norm-statistics reset based on small, non-sensitive buffers (or none if unavailable).
    • Assumptions/dependencies: Some models benefit from lightweight calibration passes; if disallowed, expect slightly smaller gains but still improved robustness compared to pruning in many cases.
  • Robust deployment on embedded and industrial devices
    • Sectors: manufacturing, automotive, drones/robotics, smart cameras/IoT
    • What to do: Compress perception models (classification/detection backbones) via folding to achieve accuracy-preserving reductions at moderate–high compression.
    • Tools/workflows: Embedded toolchain plugins (e.g., CMSIS-NN, TFLite Micro) that apply folding offline, produce merged-filter kernels, and adapt subsequent layers.
    • Assumptions/dependencies: Folding modifies topology by merging channels/filters and changing the next layer’s weights—ensure toolchain supports such rewrites.
  • OTA updates with reduced model size and better accuracy retention
    • Sectors: automotive, consumer electronics, telecom
    • What to do: Fold existing deployed models to reduce update package sizes; schedule brief on-device LN/BN recalibration after update to restore accuracy.
    • Tools/workflows: CI/CD stage that folds the model before packaging; device-side post-install calibration hook.
    • Assumptions/dependencies: Ensure devices can run a short calibration pass or accept slightly lower accuracy if disallowed.
  • A/B compression selection based on projection geometry
    • Sectors: MLOps, platform teams
    • What to do: For each model/layer, compute folding and pruning candidates and rank by reconstruction error (Frobenius norm) and post-compression validation accuracy. Choose the method that minimizes function perturbation.
    • Tools/workflows: “FoldScore” metric (parameter reconstruction error) and accuracy gate; auto-selection per layer; logs and dashboards integrated with W&B or similar.
    • Assumptions/dependencies: Gains depend on alignment between parameter vectors and training regime (learning rate, SAM, augmentation).
  • Teaching and reproducible research
    • Sectors: academia
    • What to do: Use the released code and >1,000-checkpoint study to teach projection geometry of compression, run labs on folding vs pruning, and explore how sharpness/optimizers affect compressibility.
    • Tools/workflows: Course modules with reproducible configs; lab assignments analyzing Lipschitz constants, sharpness, and parameter-space projections.
    • Assumptions/dependencies: Results clearest for CNNs/ViTs and small LLaMAs; attention-layer folding remains open.
  • Sustainability metrics in procurement and deployment
    • Sectors: policy, public procurement, sustainability offices
    • What to do: Include geometry-aware, calibration-free compression (e.g., folding) in procurement checklists for AI systems, aiming for lower energy footprints without retraining requirements.
    • Tools/workflows: Policy templates requiring evaluation of compression methods at matched budgets and reporting of energy/latency/accuracy trade-offs.
    • Assumptions/dependencies: Compression impact should be measured across representative workloads and compression levels; corner cases may narrow folding’s edge.
  • AutoML compression recipes for limited fine-tuning budgets
    • Sectors: software platforms, startups
    • What to do: Default to folding when only LayerNorm/BatchNorm reset and ≤5 fine-tuning epochs are available; folding tends to recover faster and retain an advantage over pruning under light adaptation.
    • Tools/workflows: “FoldTune” preset: fold, reset norms, fine-tune for 1–5 epochs; fallback to pruning only if validation shows reversals at very low compression.
    • Assumptions/dependencies: Effect sizes vary with hyperparameters (SAM, LR, augmentation); validate per model family.

Long-Term Applications

These items require additional research, scaling, or ecosystem support before routine deployment.

  • Folding for attention blocks and large-scale LLMs/VLMs
    • Sectors: software, healthcare, finance, education
    • What to aim for: Extend folding beyond FFNs to attention heads and multi-head projections; validate on larger LLMs and multimodal models.
    • Tools/products: Compiler and framework support to merge attention heads and adapt subsequent projections; evaluation suites for perplexity/accuracy/robustness at scale.
    • Assumptions/dependencies: Need new algorithms for head merging without degrading attention diversity; hardware-aware re-mapping.
  • Hybrid compression stacks (folding + quantization + distillation)
    • Sectors: cloud/edge AI, mobile, robotics
    • What to aim for: Combine folding’s geometry-aware projection with post-training quantization and lightweight distillation for maximal compression at minimal accuracy loss.
    • Tools/products: Layer-wise pipelines (fold → quantize → optional distill), compiler passes that co-optimize clusters and quantization scales.
    • Assumptions/dependencies: Interactions between methods need systematic study; calibration data may be required for the distillation step.
  • Training-time “foldability” regularization
    • Sectors: research, foundation-model providers
    • What to aim for: Encourage directional alignment during training (e.g., cluster-friendly weight structure) so that post-training folding becomes nearly lossless.
    • Tools/products: Regularizers that penalize within-cluster dispersion, SAM variants with folding-aware neighborhoods.
    • Assumptions/dependencies: Must balance generalization with structure; avoid entrenching biases that harm downstream tasks.
  • Dynamic or on-device adaptive folding
    • Sectors: mobile, IoT, robotics
    • What to aim for: Adjust the number of clusters at runtime based on device temperature, battery, or latency targets—graceful degradation with controllable accuracy.
    • Tools/products: Runtime controllers (e.g., governors) that swap in pre-folded subgraphs; APIs to toggle cluster counts.
    • Assumptions/dependencies: Requires multiple pre-built variants or fast on-device reconfiguration; careful state management for norms and caches.
  • Cross-layer and graph-aware folding
    • Sectors: research, compilers
    • What to aim for: Fold across layers or blocks to capture global structure (e.g., residual paths, bottlenecks) and potentially unlock larger gains than layer-wise folding.
    • Tools/products: Graph clustering passes in compilers (TVM/TensorRT) that co-optimize folding with layout and scheduling.
    • Assumptions/dependencies: Non-local coupling complicates adaptation of downstream layers; more complex validation is needed.
  • Formal safety, robustness, and fairness audits under compression
    • Sectors: policy, regulated industries
    • What to aim for: Standardize geometry-aware metrics (e.g., parameter reconstruction error, local Lipschitz estimates) in model cards and audits; certify compression-induced perturbations.
    • Tools/products: Audit toolkits that compute functional perturbation bounds and produce report-ready summaries.
    • Assumptions/dependencies: Need accepted proxies for functional change beyond accuracy; sector-specific thresholds and benchmarks.
  • Hardware–algorithm co-design for folded architectures
    • Sectors: semiconductors, embedded systems
    • What to aim for: Design kernels and accelerators that exploit shared filters/channels produced by folding (e.g., reuse of partial sums, cache locality).
    • Tools/products: Library primitives for “merged-channel” convolutions/linear layers; compiler scheduling that exploits weight tying.
    • Assumptions/dependencies: Benefits depend on memory hierarchy and kernel support; requires close coordination with vendors.
  • Autotuners leveraging sharpness and projection geometry
    • Sectors: MLOps, AutoML
    • What to aim for: Use sharpness estimates and projection error to decide per-layer whether to fold or prune and how many clusters to keep.
    • Tools/products: Layer-wise controllers that optimize compression using geometry and curvature signals; policy learning for compression decisions.
    • Assumptions/dependencies: Sharpness estimation must be stable and cheap; generalizes across architectures and datasets.
  • Theoretical advances: removing the one-rank slack and broadening assumptions
    • Sectors: academia
    • What to aim for: Prove guarantees at exactly matched ranks, extend beyond ReLU+normalization settings, and connect to generalization bounds.
    • Tools/products: Open-source benchmarks and theorem-proving notebooks to validate assumptions on more architectures and tasks.
    • Assumptions/dependencies: May require new projection operators and tighter function-perturbation analyses.
  • Standards for “Green AI” compression in public procurement
    • Sectors: policy, public sector IT
    • What to aim for: Codify requirements to evaluate geometry-aware, calibration-free compression and report energy/latency gains before deployment.
    • Tools/products: Procurement templates and scorecards with mandatory compression baselines and reporting formats.
    • Assumptions/dependencies: Coordination with standards bodies; consensus on metrics and thresholds.

Notes on feasibility across applications

  • Best-performing regimes: moderate–high compression and training setups that yield flatter, structured solutions (moderate learning rates, SAM with small–moderate radius).
  • Corner cases: very low compression, very high/very low learning rates, strong augmentations, or large SAM radii can narrow or occasionally reverse folding’s advantage—validate per model.
  • Current scope: empirical results cover CNNs/ViTs and small LLaMAs (FFN blocks). Attention folding and very large LLMs require further work.
  • Compute considerations: folding adds a one-time k-means overhead; inference compute and memory match pruning at the same retained size once graphs are rebuilt.
View Paper Prompt  View All Prompts 

Glossary

  • axis-aligned projection: A projection onto axes of the coordinate system, removing components by zeroing specific coordinates. "structured pruning is an axis-aligned projection"
  • BatchNorm: A normalization technique that stabilizes training by normalizing activations within a mini-batch. "We assume ReLU activations and normalization layers (e.g. BatchNorm or LayerNorm) are present."
  • calibration-free: Compression or evaluation performed without using calibration data or training inputs. "We study calibration-free compression through the lens of projection geometry"
  • cluster-structured subspaces: Subspaces defined by grouping parameters into clusters, preserving directional information via cluster means. "folding projects onto cluster-structured subspaces that retain directional information."
  • coordinate-aligned subspace: A subspace formed by selecting specific coordinate axes, typically by keeping certain neurons/channels and removing others. "Pruning can be viewed as a projection onto a coordinate-aligned subspace at the level of neurons, filters, or channels."
  • Euclidean norm: The standard L2 norm measuring vector length in Euclidean space. "Such projections map any parameter vector to its closest point (in the Euclidean norm) within a lower-dimensional subspace."
  • FFN blocks: Feed-Forward Network blocks within transformer architectures. "We prune only FFN blocks."
  • Frobenius norm: A matrix norm equal to the square root of the sum of squared entries, equivalent to the L2 norm of the vectorized matrix. "The Frobenius norm of a matrix is defined as ∥A∥F=∑i,j∣aij∣2\lVert A \rVert_F = \sqrt{\sum_{i,j} |a_{ij}|^2}"
  • functional perturbations: Changes in the network’s output behavior due to parameter modifications. "smaller functional perturbations than pruning."
  • Hartigan’s algorithm: A classic algorithm for k-means clustering that iteratively reassigns points to improve the objective. "Using Hartigan’s algorithm~\citep{Hartigan1979}, one sweep costs O(pkm)O(pkm)"
  • idempotent: An operator that, when applied multiple times, yields the same result as applying it once. "i.e. it is symmetric and idempotent."
  • k-means clustering: A clustering method that partitions data into k groups minimizing within-cluster variance. "Let Uf⋆\mathbf{U}^\star_f be the basis obtained from an optimal kk-means clustering with kfk_f clusters"
  • LayerNorm: A normalization technique that normalizes across features within a single sample. "We assume ReLU activations and normalization layers (e.g. BatchNorm or LayerNorm) are present."
  • Lipschitz continuous: A function whose change is bounded linearly by the change in input under a constant factor. "We assume that the loss function\,LL is Lipschitz continuous"
  • magnitude-based pruning: Pruning that removes parameters based on the magnitude of weights (e.g., L1/L2 criteria). "the most widely used is magnitude-based pruning"
  • model folding: Compression that clusters similar weights and ties them together by replacing cluster members with their mean. "the recently introduced model folding clusters similar weights and ties them together"
  • orthogonal operators: Linear operators that are equal to their transpose and project onto subspaces without distortion along orthogonal complements. "We formalize both as orthogonal operators"
  • orthogonal projection: The projection onto a subspace that minimizes Euclidean distance, implemented by a symmetric, idempotent matrix. "A matrix C∈Rm×m\mathbf{C} \in \mathbb{R}^{m\times m} is an orthogonal projection if C=C⊤=C2\mathbf{C} = \mathbf{C}^\top = \mathbf{C}^2"
  • perplexity: A measure of LLM uncertainty; lower values indicate better performance. "Columns 4--8 show perplexity of the trained model before compression and after pruning / folding"
  • projection geometry: The geometric perspective that analyzes compression as projections in parameter space. "Cut Less, Fold More: Model Compression\ through the Lens of Projection Geometry"
  • projection matrix: The matrix that maps vectors onto a subspace, typically constructed from a basis. "with projection matrix Cp\mathbf{C}_p and transformed weight matrix Wp\mathbf{W}_p:"
  • rank distance: The difference in dimensionality (rank) between two compressed representations. "within a rank distance of one"
  • REPAIR: A post-processing method that re-estimates normalization statistics to improve pruned models. "for CNNs we only re-estimate batch-normalization statistics via a single forward pass using REPAIR~\citep{jordan2023repairrenormalizingpermutedactivations}"
  • sharpness-aware training (SAM): An optimization method that seeks parameters in flatter regions by penalizing loss increases under perturbations. "sharpness-aware training (SAM)"
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Tweets

Sign up for free to view the 5 tweets with 90 likes about this paper.

Sign Up for Free