Post-Training Statistical Calibration for Higher Activation Sparsity (2412.07174v1)

Abstract: We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: https://github.com/IntelLabs/SCAP.

• The paper introduces SCAP as a post-training method that prunes input activations, achieving 48.5% FFN sparsity compared to 33.3% with prior approaches.
• It employs a novel mode-centering technique that increases input sparsity by 44.7% in non-GLU Transformer architectures to enhance decoding efficiency.
• The framework demonstrates versatility across diverse Transformer models, highlighting its potential for resource-efficient deployment in real-world AI applications.

An Evaluation of Statistical Calibrated Activation Pruning (SCAP) for Enhanced Activation Sparsity in Transformers

The paper introduces Statistical Calibrated Activation Pruning (SCAP) as an innovative framework for post-training activation pruning that targets the sparsification of input activations within Fully-Connected (FC) layers in Transformer architectures. The authors argue that SCAP provides a significant improvement over existing models, particularly with respect to LLMs decoding latency, exemplifying its utility in increasing activation sparsity without adverse impacts on model performance.

Key Contributions and Empirical Findings

1. Generalized Activation Pruning: SCAP's primary enhancement over existing methodologies like CATS lies in its generalized approach to pruning. By focusing on input activations rather than the outputs of activation functions, SCAP leverages the sparsity in input distributions, achieving a Pareto-efficient trade-off between task performance and computational efficiency. Empirical evidence is presented where SCAP achieves an FFN sparsity of 48.5%, compared to CATS' 33.3%, reflecting a 27.1% enhancement in decoding speed over the dense model.
2. Mode-Centering Technique: A novel Mode-Centering pre-calibration is proposed to address skewed activation distributions that impede pruning potential. This calibration method maximizes sparsity by re-centering the mode of input activations, significantly increasing input sparsity, particularly in non-GLU Transformer architectures. Results showcased a 44.7% increase in input sparsity for MPT-7B, emphasizing the utility of Mode-Centering in enhancing sparsity extraction.
3. Extensive Model Coverage: The application scope of SCAP extends across a variety of model architectures, including pre-quantized Transformers and emerging Transformer types such as Mamba2, substantiating the framework's versatility and practical utility in real-world deployments. The consistent performance of SCAP across these varied architectures underlines its scalability and efficacy in LLM deployment scenarios.

Implications for Future Developments

SCAP's approach of enhancing activation sparsity presents an avenue for efficiency gains in machine learning model deployment, particularly in scenarios constrained by memory bandwidth bottlenecks. By reducing the memory requirement through intelligent sparsification, SCAP could potentially facilitate more widespread adoption of sophisticated LLMs on consumer-grade hardware, democratizing access to advanced AI systems. The empirical evidence supporting SCAP's improved decoding speeds could also encourage further exploration of decoupling sparsity mechanisms across different Transformer layers for optimized performance.

Future research could focus on expanding the scope of SCAP through parameter-efficient fine-tuning enhancements or exploring its synergy with efficient attention mechanisms to counteract the diminishing returns in speedup observed with longer prompts. The successful integration of SCAP with diverse Transformer types besides LLMs also opens up possibilities for its application in domains beyond natural language processing, such as vision and multi-modal models.

Overall, SCAP presents a potent approach for achieving activation sparsity in Transformer models, optimizing both performance and computational efficiency with minimal retraining requirements. This capability has significant implications, making SCAP a promising addition to post-training sparsity techniques within the rapidly evolving field of AI.