Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Chunk-Distilled Language Modeling (2501.00343v1)

Published 31 Dec 2024 in cs.CL and cs.AI

Abstract: We introduce Chunk-Distilled LLMing (CD-LM), an approach to text generation that addresses two challenges in current LLMs: the inefficiency of token-level generation, and the difficulty of adapting to new data and knowledge. Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step. Our retrieval framework enables flexible construction of model- or domain-specific datastores, either leveraging the internal knowledge of existing models, or incorporating expert insights from human-annotated corpora. This adaptability allows for enhanced control over the LLM's distribution without necessitating additional training. We present the CD-LM formulation along with performance metrics demonstrating its ability to improve LLM performance and efficiency across a diverse set of downstream tasks. Code and data will be made publicly available.

Summary

We haven't generated a summary for this paper yet.

Youtube Logo Streamline Icon: https://streamlinehq.com