Resonance RoPE: Improving Context Length Generalization of Large Language Models (2403.00071v2)
Abstract: This paper addresses the challenge of train-short-test-long (TSTL) scenarios in LLMs equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream LLMing tasks and a variety of downstream long-text applications.
- L-eval: Instituting standardized evaluation for long context language models. CoRR, abs/2307.11088.
- Zhangir Azerbayev. 2022. zhangir-azerbayev/proof-pile.
- Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308.14508.
- bloc97. 2023. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.
- Extending context window of large language models via positional interpolation. CoRR, abs/2306.15595.
- Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691.
- Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436. Association for Computational Linguistics.
- Mistral 7b. CoRR, abs/2310.06825.
- Mixtral of experts. CoRR, abs/2401.04088.
- The impact of positional encoding on length generalization in transformers. CoRR, abs/2305.19466.
- Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations. OpenReview.net.
- Scaling laws of rope-based extrapolation. CoRR, abs/2310.05209.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations. OpenReview.net.
- Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations. OpenReview.net.
- OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
- Yarn: Efficient context window extension of large language models. CoRR, abs/2309.00071.
- Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations. OpenReview.net.
- Compressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations. OpenReview.net.
- Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
- Zero-offload: Democratizing billion-scale model training. In 2021 USENIX Annual Technical Conference, pages 551–564. USENIX Association.
- Code llama: Open foundation models for code. CoRR, abs/2308.12950.
- Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1889–1903, Toronto, Canada. Association for Computational Linguistics.
- ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
- Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
- Long range arena : A benchmark for efficient transformers. In 9th International Conference on Learning Representations. OpenReview.net.
- Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
- Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.
- Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008.
- Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022.
- Memorizing transformers. In The Tenth International Conference on Learning Representations. OpenReview.net.
- Effective long-context scaling of foundation models. CoRR, abs/2309.16039.
- Length extrapolation of transformers: A survey from the perspective of position encoding. CoRR, abs/2312.17044.
- Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations. OpenReview.net.
- Pose: Efficient context window extension of llms via positional skip-wise training. CoRR, abs/2309.10400.
- Suyuchen Wang (16 papers)
- Ivan Kobyzev (23 papers)
- Peng Lu (86 papers)
- Mehdi Rezagholizadeh (78 papers)
- Bang Liu (93 papers)