Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Compress Prompt in Natural Language Formats (2402.18700v2)

Published 28 Feb 2024 in cs.CL, cs.AI, and cs.LG
Learning to Compress Prompt in Natural Language Formats

Abstract: LLMs are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.

PDF HTML Abstract

Learning to Compress Prompt in Natural Language Formats

The paper entitled "Learning to Compress Prompt in Natural Language Formats" explores the challenges and solutions associated with reducing the length of prompts used in LLMs while maintaining their effectiveness. The authors introduce an innovative framework, referred to as Nano-Capsulator, that employs a novel compression technique aimed at converting long prompts into shorter, natural language (NL) formatted prompts, termed Capsules. This approach addresses the two primary issues faced by existing soft prompt compression methods: transferability and flexibility across different LLMs.

Key Contributions

The main contributions of this paper are multi-fold:

  • Framework Introduction: The authors propose the Nano-Capsulator framework, which involves compressing long prompts into NL Capsules. These Capsules retain a high degree of semantic relevance and offer better transferability across different LLMs and datasets.
  • Optimization Techniques: The compression is achieved by employing a semantic preservation loss and a reward-based optimization to maintain the utility of the compressed prompts.
  • Practical Benefits: Experimental results reveal that the Capsule can significantly reduce the length of the original prompts by up to 81.4%, decrease inference latency by as much as 4.5 times, and cut budget overheads by 80.1% while preserving performance.

Methodology

The compression is guided by a well-structured optimization process that ensures both semantic fidelity and task utility. The reward-based optimization considers task-specific question-answer pairs to fine-tune the Capsules, maintaining the effectiveness of the prompts under length constraints. The overall loss function integrates a semantic preservation component to ensure the shorter prompts retain the essence of the longer ones.

Experimental Results and Implications

Evaluation Metrics and Datasets

The framework was tested on multiple datasets and LLMs to validate its effectiveness:

  • Few-shot CoT: Using datasets such as CommonsenseQA (CSQA) and GSM8K.
  • Reading Comprehension: Evaluations were conducted on MultiRC and TriviaQA-Long datasets.

The performance metrics include accuracy for individual tasks along with compression rate, latency reduction, and cost savings.

Key Findings

  • Effective Compression: Capsules achieve substantial compression rates while retaining similar performance levels across different LLMs such as Vicuna-13B, PaLM, and Claude2.
  • Reduced Cost and Latency: Significant reductions in computational costs and latency were observed. For instance, on the Claude2 API, Capsules saved up to 80.1% of the cost and reduced inference latency by up to 4.5 times.
  • High Transferability: The framework showed strong results even when applied to unseen datasets, suggesting that the approach generalizes well across different domains without requiring retraining.

Discussion

The implications of these findings are significant in both theoretical and practical contexts. Theoretically, this work underscores the potential for improving LLM efficiency without substantial performance trade-offs. Practically, the reduction in computational overhead and cost makes it more feasible to deploy LLMs at scale, especially in industries where cost and speed are critical factors. The ability to maintain performance across various LLMs and datasets also highlights the robustness of the Nano-Capsulator framework.

Looking forward, future research could explore further optimization techniques and adaptations of the Nano-Capsulator framework to extend its applicability to more diverse types of LLM tasks and larger-scale datasets. Integrating this framework with cross-modal architectures such as those involving vision and language tasks could also be a fruitful direction.

Conclusion

The proposed Nano-Capsulator framework showcases a promising methodology for prompt compression in LLMs, addressing key challenges in transferability and computational efficiency. The results demonstrate substantial benefits in reducing prompt lengths, cutting costs, and decreasing latency—all while preserving the utility and effectiveness of the prompts. This work marks an important step towards more practical and scalable applications of LLMs, paving the way for broader adoption and use-case diversity. Given its robust performance and applicability, the Nano-Capsulator framework sets a foundational precedent for future advancements in prompt optimization and compression strategies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. Anthropic. 2023. Claude.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Adapting language models to compress contexts. arXiv preprint arXiv:2305.14788.
  4. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  5. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  6. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  7. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  8. Eraser: A benchmark to evaluate rationalized nlp models.
  9. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945.
  10. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736.
  11. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325.
  12. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, page arXiv:1705.03551.
  13. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
  14. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  15. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201.
  16. G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023. arXiv preprint arXiv:2303.16634, 6.
  17. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467.
  18. Context compression for auto-regressive transformers with sentinel tokens. arXiv preprint arXiv:2310.08152.
  19. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  20. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  21. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  22. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  23. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. arXiv preprint arXiv:2210.03162.
  24. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712.
  25. Opt: Open pre-trained transformer language models.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yu-Neng Chuang (28 papers)
  2. Tianwei Xing (7 papers)
  3. Chia-Yuan Chang (18 papers)
  4. Zirui Liu (58 papers)
  5. Xun Chen (166 papers)
  6. Xia Hu (186 papers)
Citations (12)
View on Semantic Scholar