BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
Abstract: LLMs have revolutionized numerous applications, yet their deployment remains challenged by memory constraints on local devices. While scaling laws have enhanced LLM capabilities, the primary bottleneck has shifted from \textit{capability} to \textit{availability}, emphasizing the need for efficient memory management. Traditional compression methods, such as quantization, often require predefined compression ratios and separate compression processes for each setting, complicating deployment in variable memory environments. In this paper, we introduce \textbf{BitStack}, a novel, training-free weight compression approach that enables megabyte-level trade-offs between memory usage and model performance. By leveraging weight decomposition, BitStack can dynamically adjust the model size with minimal transmission between running memory and storage devices. Our approach iteratively decomposes weight matrices while considering the significance of each parameter, resulting in an approximately 1-bit per parameter residual block in each decomposition iteration. These blocks are sorted and stacked in storage as basic transmission units, with different quantities loaded based on current memory availability. Extensive experiments across a wide range of tasks demonstrate that, despite offering fine-grained size control, BitStack consistently matches or surpasses strong quantization baselines, particularly at extreme compression ratios. To the best of our knowledge, this is the first decomposition-based method that effectively bridges the gap to practical compression techniques like quantization. Code is available at https://github.com/xinghaow99/BitStack.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- Anthropic. Claude ai. https://www.anthropic.com/claude, 2024. Accessed: 2024-09-15.
- Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024.
- Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020.
- Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
- Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332, 2022.
- The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
- Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118, 2024.
- Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
- A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.org/records/12608602.
- GitHub. Github copilot. https://github.com/features/copilot, 2024. Accessed: 2024-09-15.
- Google. Google gemini ai. https://gemini.google.com/, 2024. Accessed: 2024-09-15.
- Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022.
- Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291, 2024a.
- How good are low-bit quantized llama3 models? an empirical study. arXiv preprint arXiv:2404.14047, 2024b.
- Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
- Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. URL https://arxiv.org/abs/2306.00978.
- Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023.
- Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
- Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679, 2024.
- OpenAI. Chatgpt. https://chat.openai.com/, 2024. Accessed: 2024-09-15.
- Perplexity.AI. Perplexity ai. https://www.perplexity.ai/, 2024. Accessed: 2024-09-15.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.
- Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
- Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024.
- Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024.
- Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023.
- Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295, 2024.
- Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
- Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
- Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.