Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation (2401.16421v2)

Published 29 Jan 2024 in cs.LG, cs.AI, cs.CL, and stat.ML

Abstract: In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. CoLT5: Faster long-range transformers with conditional computation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  2. Communicating hierarchical state machines. In Automata, Languages and Programming: 26th International Colloquium, ICALP’99 Prague, Czech Republic, July 11–15, 1999 Proceedings 26, pp.  169–178. Springer, 1999.
  3. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
  4. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450, 2023a.
  7. SummScreen: A dataset for abstractive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  8602–8615, May 2022.
  8. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023b.
  9. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399, 2022.
  10. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  13522–13537, 2023.
  11. Palm: Scaling language modeling with pathways, 2022.
  12. Monotonic location attention for length generalization. arXiv preprint arXiv:2305.20019, 2023.
  13. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704, 2016.
  14. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2021.
  15. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), June 2019a.
  16. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.  4171–4186, June 2019b.
  17. Eilenberg, S. Automata, languages, and machines. Academic press, 1974.
  18. Towards revealing the mystery behind chain of thought: A theoretical perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  19. The hierarchical hidden markov model: Analysis and applications. Machine learning, 32:41–62, 1998.
  20. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
  21. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836.
  22. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16, 2003.
  23. Halliday’s introduction to functional grammar. Routledge, 2013.
  24. Lm-infinite: Simple on-the-fly length generalization for large language models, 2023.
  25. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.  1382–1390, December 2022.
  26. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  27. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  1419–1436, June 2021.
  28. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325, 2024.
  29. The impact of positional encoding on length generalization in transformers. arXiv preprint arXiv:2305.19466, 2023.
  30. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6, 2018.
  31. ContractNLI: A dataset for document-level natural language inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp.  1907–1919, November 2021.
  32. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.  785–794, September 2017.
  33. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023.
  34. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  3214–3252, May 2022.
  35. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749, 2022.
  36. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023.
  37. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  5336–5358, July 2022.
  38. Yarn: Efficient context window extension of large language models, 2023.
  39. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations (ICLR), 2022.
  40. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations (ICLR), 2020a.
  41. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020b.
  42. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  43. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  6383–6402, 2023.
  44. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  45. Randomized positional encodings boost length generalization of transformers. In Association for Computational Linguistics (ACL), July 2023a.
  46. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.  1889–1903, July 2023b.
  47. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.  8732–8740, 2020.
  48. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533, 2022.
  49. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018.
  50. Roformer: Enhanced transformer with rotary position embedding, 2021.
  51. A length-extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, July 2023.
  52. Llama 2: Open foundation and fine-tuned chat models, 2023.
  53. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
  54. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
  55. Efficient streaming language models with attention sinks, 2023.
  56. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  57. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  5905–5921, June 2021.
  58. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Zhenyu He (57 papers)
  2. Guhao Feng (8 papers)
  3. Shengjie Luo (20 papers)
  4. Kai Yang (187 papers)
  5. Di He (108 papers)
  6. Jingjing Xu (80 papers)
  7. Zhi Zhang (113 papers)
  8. Hongxia Yang (130 papers)
  9. Liwei Wang (239 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets