Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
72 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference (2405.17755v1)

Published 28 May 2024 in cs.CL and cs.AI

Abstract: Length generalization failure problem, namely the LLM fails to generalize to texts longer than its maximum training length, greatly restricts the application of LLM in the scenarios with streaming long inputs. To address this problem, the existing methods either require substantial costs or introduce precision loss. In this paper, we empirically find that the accuracy of the LLM's prediction is highly correlated to its certainty. Based on this, we propose an efficient training free framework, named XL3M (it means extra-long LLM), which enables the LLMs trained on short sequences to reason extremely long sequence without any further training or fine-tuning. Under the XL3M framework, the input context will be firstly decomposed into multiple short sub-contexts, where each sub-context contains an independent segment and a common question'' which is a few tokens from the end of the original context. Then XL3M gives a method to measure the relevance between each segment and thequestion'', and constructs a concise key context by splicing all the relevant segments in chronological order. The key context is further used instead of the original context to complete the inference task. Evaluations on comprehensive benchmarks show the superiority of XL3M. Using our framework, a Llama2-7B model is able to reason 20M long sequences on an 8-card Huawei Ascend 910B NPU machine with 64GB memory per card.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
  2. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.
  3. Unlimiformer: Long-range transformers with unlimited length input. Advances in Neural Information Processing Systems, 36, 2024.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023.
  6. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023.
  7. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
  8. The impact of positional encoding on length generalization in transformers. arXiv preprint arXiv:2305.19466, 2023.
  9. How long can open-source llms truly promise on context length, 2023.
  10. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024.
  11. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435, 2023.
  12. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308.10882, 2023.
  13. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023.
  14. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021.
  15. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  16. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  6383–6402, 2023.
  17. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  18. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
  19. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  20. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023.
  21. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  22. An overview on language models: Recent developments and outlook. arXiv preprint arXiv:2303.05759, 2023.
  23. Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022.
  24. Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv preprint arXiv:2402.04617, 2024.
  25. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023.
  26. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023.
  27. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Shengnan Wang (12 papers)
  2. Youhui Bai (4 papers)
  3. Lin Zhang (342 papers)
  4. Pingyi Zhou (9 papers)
  5. Shixiong Zhao (7 papers)
  6. Gong Zhang (38 papers)
  7. Sen Wang (164 papers)
  8. Renhai Chen (4 papers)
  9. Hua Xu (78 papers)
  10. Hongwei Sun (14 papers)
Citations (2)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets