Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression (2402.16058v1)

Published 25 Feb 2024 in cs.CL

Abstract: LLMs require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based LLM and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla LLM. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding LLMs. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Gpt-4 technical report.
  2. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, (PLDI):1946–1969.
  3. Language models are few-shot learners. In Proceedings of NeurIPS.
  4. Unleashing the potential of prompt engineering in large language models: a comprehensive review. ArXiv preprint.
  5. Black-box prompt optimization: Aligning large language models without model training. ArXiv preprint.
  6. Adapting language models to compress contexts. In Proceedings of EMNLP, pages 3829–3846.
  7. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  8. Scaling instruction-finetuned language models.
  9. Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions?
  10. In-context autoencoder for context compression in a large language model.
  11. Peter D Grünwald. 2007. The minimum description length principle.
  12. Retrieval augmented language model pre-training. In Proceedings of ICML, pages 3929–3938.
  13. Codecot and beyond: Learning to program and test like a developer.
  14. Few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24:251:1–251:43.
  15. Can large language models truly understand prompts? a case study with negated prompts. In Transfer Learning for Natural Language Processing Workshop, pages 52–62. PMLR.
  16. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL, pages 1601–1611.
  17. Challenges and applications of large language models.
  18. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. In Proceedings of EMNLP, pages 1813–1829.
  19. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, pages 452–466.
  20. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL, pages 7871–7880.
  21. Structured chain-of-thought prompting for code generation.
  22. Yucheng Li. 2023. Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering.
  23. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, (9):1–35.
  24. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of ACL, pages 8086–8098.
  25. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of ACL, pages 9802–9822.
  26. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of EMNLP, pages 11048–11064.
  27. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of ACL, pages 3470–3487.
  28. Learning to compress prompts with gist tokens.
  29. Ms marco: A human-generated machine reading comprehension dataset. In CoCo@ NIPs.
  30. OpenAI. 2022. Chatgpt.
  31. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, pages 27730–27744.
  32. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523–2544.
  33. In-context retrieval-augmented language models.
  34. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
  35. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, (3):379–423.
  36. Replug: Retrieval-augmented black-box language models.
  37. Learning by distilling context.
  38. Stanford alpaca: An instruction-following llama model.
  39. A comprehensive survey of hallucination mitigation techniques in large language models.
  40. Llama: Open and efficient foundation language models.
  41. Attention is all you need. In Proceedings of NeurIPS, pages 5998–6008.
  42. Exploring neural models for query-focused summarization. In Findings of the Association for Computational Linguistics: NAACL, pages 1455–1468.
  43. Explore-instruct: Enhancing domain-specific instruction coverage through active exploration. In Proceedings of EMNLP, pages 9435–9454.
  44. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of ACL, pages 13484–13508.
  45. Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of EMNLP, pages 5085–5109.
  46. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR.
  47. Emergent abilities of large language models. Transactions on Machine Learning Research.
  48. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, pages 24824–24837.
  49. Huggingface’s transformers: State-of-the-art natural language processing.
  50. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of ACL, pages 1423–1436.
  51. Recomp: Improving retrieval-augmented lms with compression and selective augmentation.
  52. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, pages 2369–2380.
  53. Investigating the effectiveness of task-agnostic prefix prompt for instruction following. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
  54. Openmatch-v2: An all-in-one multi-modality plm-based information retrieval toolkit. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3160–3164.
  55. Augmentation-adapted retriever improves generalization of language models as generic plug-in. In Proceedings of ACL, pages 2421–2436.
  56. A survey of large language models.
  57. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Xinze Li (34 papers)
  2. Zhenghao Liu (77 papers)
  3. Chenyan Xiong (95 papers)
  4. Shi Yu (37 papers)
  5. Yukun Yan (39 papers)
  6. Shuo Wang (382 papers)
  7. Ge Yu (63 papers)
Citations (3)
View on Semantic Scholar