Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (2404.10308v1)
Abstract: LLMs have shown remarkable performance in various natural language processing tasks. However, a primary constraint they face is the context limit, i.e., the maximum number of tokens they can process. Previous works have explored architectural changes and modifications in positional encoding to relax the constraint, but they often require expensive training or do not address the computational demands of self-attention. In this paper, we present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations. HOMER uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. Each chunk is then processed collectively, employing a hierarchical strategy that merges adjacent chunks at progressive transformer layers. A token reduction technique precedes each merging, ensuring memory usage efficiency. We also propose an optimized computational order reducing the memory requirement to logarithmically scale with respect to input length, making it especially favorable for environments with tight memory restrictions. Our experiments demonstrate the proposed method's superior performance and memory efficiency, enabling the broader use of LLMs in contexts requiring extended context. Code is available at https://github.com/alinlab/HOMER.
- Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
- Unlimiformer: Long-range transformers with unlimited length input. arXiv preprint arXiv:2305.01625, 2023.
- bloc97. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_%20scaled_rope_allows_llama_models_to_have/, 2023.
- Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
- Carol Chen. Transformer inference arithmetic. https://kipp.ly/blog/transformer-inference-arithmetic/, 2022.
- Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023.
- Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
- Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2021.
- Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
- Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
- An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
- Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023.
- Which tokens to use? investigating token reduction in vision transformers. arXiv preprint arXiv:2308.04657, 2023.
- Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284–299, 2023.
- Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020.
- kaiokendev. Things i’m learning while training superhot. https://kaiokendev.github.io/til#extending-context-to-8k./, 2023.
- Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, 2020.
- Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
- Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022.
- Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
- Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023.
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608, 2021.
- Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023.
- Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972, 2019.
- Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
- Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196, 2023.
- Jianlin Su. Rectified rotary position embeddings. https://github.com/bojone/rerope, 2023.
- Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
- Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
- Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
- Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023.
- Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022.
- Big bird: Transformers for longer sequences. Neural Information Processing Systems, 2020.
- Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697–12706. PMLR, 2021.
- Woomin Song (6 papers)
- Seunghyuk Oh (5 papers)
- Sangwoo Mo (20 papers)
- Jaehyung Kim (44 papers)
- Sukmin Yun (10 papers)
- Jung-Woo Ha (67 papers)
- Jinwoo Shin (196 papers)