EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression

Abstract: Post-Training Sparsity (PTS) has emerged as a crucial paradigm for compressing LLMs to facilitate efficient deployment on resource-constrained devices. However, existing PTS methodologies are typically confined to Single-Sparsity optimization, necessitating a separate, time-consuming optimization session for each specific sparsity level. This rigid paradigm significantly hinders flexible deployment across diverse hardware scenarios, as adapting to a new sparsity requirement mandates a complete re-optimization process. To address these limitations, we propose Elastic Post-Training Sparsity (EPTS), a unified Multi-Sparsity framework that produces a single elastic model capable of maintaining robust performance across diverse sparsity configurations through a one-shot optimization process. Specifically, we design a Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) mechanism that facilitates knowledge inheritance from low- to high-sparsity groups, effectively mitigating the competition for parameter reconstruction. Furthermore, we introduce a Multi-Sparsity Feature Mixer (MSFM), which significantly enhances the model's adaptability to pruning perturbations by dynamically fusing feature representations of varying sparsity granularities. Extensive experiments on LLaMA and OPT families demonstrate that EPTS achieves competitive performance compared to state-of-the-art methods like SparseGPT and Wanda, while offering significant efficiency gains by enabling multi-scenario deployment from a single optimization. our source code is available at https://github.com/xuke225/EPTS.

• The paper presents EPTS, which achieves efficient LLM compression by enabling dynamic sparsity selection via unified one-shot optimization.
• It introduces MS-HiLoRA and MSFM modules to mitigate parameter competition and adaptively fuse features across varying sparsity levels.
• Experimental results on LLaMA and OPT models demonstrate lower perplexity and 2.31–7.62% higher zero-shot accuracy at 70% sparsity.

EPTS: Elastic Post-Training Sparsity for Efficient LLM Compression

Introduction

LLMs present substantial computational demands that impede their deployment on resource-constrained hardware. Model pruning has become a standard paradigm for alleviating these requirements by inducing sparse structures in weight matrices. However, existing Post-Training Sparsity (PTS) methods predominantly optimize for a single, fixed sparsity ratio, incurring high overhead and flexibility limitations when adapting models to different deployment conditions. The paper "EPTS: Elastic Post-Training Sparsity for Efficient LLM Compression" (2606.25285) introduces a unified framework built to address these bottlenecks by producing an elastic model robust to a continuum of sparsity ratios in a single reconstruction pass.

Core Contributions

EPTS makes three central technical advances over prior PTS formulations:

1. Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) introduces a hierarchical organization of LoRA-based compensation modules. This design enforces parameter inheritance from low- to high-sparsity groups, mitigating parameter competition and supporting robust feature restoration across the sparsity range.
2. Multi-Sparsity Feature Mixer (MSFM) enables deterministic, adaptive fusion of representations generated under different sparsity granularities, thereby enhancing resilience to activation distribution shifts and sparsity noise across the network.
3. Unified One-Shot Elastic Optimization allows for training a single model capable of dynamic sparsity selection at inference, markedly reducing the cumulative time and data requirements associated with obtaining multiple sparsity-specific variants.

These mechanisms are formalized with explicit objectives that minimize reconstruction error across sampled sparsity ratios, leveraging cumulative LoRA residuals with asymmetric gradient flow properties, and strategically fusing multi-sparsity features.

Experimental Validation

EPTS is exhaustively validated on LLaMA and OPT families, with models ranging up to 8B parameters. The evaluation protocol includes perplexity (WikiText2) and zero-shot generalization across a diverse set of tasks (BoolQ, RTE, HellaSwag, WinoGrande, ARC-Challenge/Easy, OpenBookQA).

Numerical Results

Key findings:

• At 70% sparsity on LLaMA-7B, EPTS achieves a perplexity of 16.94 versus 25.78 (SparseGPT), 82.19 (Wanda), and 91.23 (RIA). On OPT-125M, EPTS scores 64.90 against 223.53 (SparseGPT), 331.00 (Wanda), and 353.96 (RIA).
• EPTS yields 2.31–7.62% higher average zero-shot accuracy than baselines at 70% sparsity across diverse NLP tasks.
• Under severe compression, optimization-free methods collapse (perplexity $>200$), while EPTS maintains robust output distribution fidelity.

Ablations confirm that MS-HiLoRA with hierarchical parameterization outperforms both dense sharing and hard separation, particularly at high sparsity. The MSFM further reduces distribution shift, leading to lower perplexity and higher accuracy at elevated pruning rates.

Time and sample efficiency are also addressed: EPTS can achieve competitive compression with as few as 32 calibration samples and converges in 1–2 reconstruction epochs for moderate sparsity. Unlike per-target optimization baselines, elastic sparsity enables deployment from a single reconstruction run.

Theoretical and Practical Implications

EPTS reframes PTS as a multi-task, hierarchical compensation problem, formalized by the Nested Information Loss Hypothesis: compensation for high-sparsity regimes must subsume and extend restoration learned for lower sparsities. The asymmetric gradient structure ensures base modules capture robust low-sparsity features, while higher-level modules specialize for aggressive pruning.

Practically, elastic sparsity unifies edge and server deployment pathways, dramatically amortizing optimization overheads and facilitating instant adaptation to real-time hardware constraints. The deterministic, block-wise optimization and MSFM fusion sidestep out-of-memory issues endemic to global optimization and enable support for non-uniform, per-layer sparsity profiles.

Future Directions and Limitations

EPTS exposes several open challenges:

• Extremely High Sparsity: The capacity for robust feature restoration degrades significantly above 80% sparsity, suggesting diminishing returns without structural or semi-structured pruning integration.
• Scalability: While designed with scalability in mind, validation on models above 8B remains logistically challenging but is theoretically supported by empirical scaling laws observed in LLM compression.

Possible research extensions include hybridizing EPTS with structured sparsification and further exploring the joint optimization landscapes over sparsity, quantization, and distillation for large-scale foundation models.

Conclusion

EPTS provides a rigorous, unified approach to post-training sparsity for LLMs, supporting flexible compression through hierarchical LoRA parameterization and deterministic multi-sparsity feature fusion. It extends the practical applicability of PTS methods to real-world, multi-scenario deployment, substantially improving both effectiveness and efficiency within a principled framework. The implications for AI deployment are significant, paving the way for resource-adaptive inference without quality compromise across hardware scales (2606.25285).