Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models (2506.00773v2)

Abstract: LLMs often struggle to accurately read and comprehend extremely long texts. Current methods for improvement typically rely on splitting long contexts into fixed-length chunks. However, fixed truncation risks separating semantically relevant content, leading to ambiguity and compromising accurate understanding. To overcome this limitation, we propose a straightforward approach for dynamically separating and selecting chunks of long context, facilitating a more streamlined input for LLMs. In particular, we compute semantic similarities between adjacent sentences, using lower similarities to adaptively divide long contexts into variable-length chunks. We further train a question-aware classifier to select sensitive chunks that are critical for answering specific questions. Experimental results on both single-hop and multi-hop question-answering benchmarks show that the proposed approach consistently outperforms strong baselines. Notably, it maintains robustness across a wide range of input lengths, handling sequences of up to 256k tokens. Our datasets and code are available at the following link: https://github.com/ECNU-Text-Computing/DCS

Overview of "Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in LLMs"

The paper "Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in LLMs" addresses a significant challenge in the field of natural language processing, specifically the difficulty LLMs encounter when processing ultra-long texts. Traditional methods generally rely on dividing long contexts into fixed-length chunks, which often disrupts the semantic flow of the text and negatively impacts the model's ability to comprehend and answer questions accurately. This paper proposes an innovative approach that dynamically segments text into semantically coherent chunks and utilizes a question-aware classifier to effectively select relevant chunks for question answering tasks.

Methodology

The proposed method, termed Dynamic Chunking and Selection (DCS), employs a multi-step process to enhance LLMs' ability to handle long texts:

1. Dynamic Chunking:
   • The authors utilize Sentence-BERT to encode sentences of a lengthy context, calculating semantic similarities between adjacent sentences. This allows for the dynamic segmentation of text into variable-length chunks, preserving semantic coherence better than fixed-length chunking methods. The chunk length is adjusted based on cosine similarity, which identifies semantic boundaries effectively.
2. Chunk Selection:
   • A question-aware classifier is trained to evaluate and select relevant chunks. This classifier uses strategically selected states from the LLM’s final transformer layer to minimize complexity and improve efficiency. By focusing on question-sensitive information, this step ensures that the LLM is presented with the most pertinent data for generating accurate responses.
3. Evaluation and Robustness:
   • The approach was tested on twelve diverse long-context reading comprehension datasets, including both single-hop and multi-hop question-answering tasks. The proposed method consistently outperformed strong baselines, demonstrating robust performance across varying input lengths, up to 256k tokens.

Results and Implications

DCS exhibits notable improvements over existing state-of-the-art methods across multiple benchmarks. The experimental findings underline the method's robustness and scalability, particularly in handling extended context lengths without a substantial loss in performance. This robustness is crucial for practical applications in tasks that require understanding extensive texts, such as legal document analysis or scientific literature review.

The implications of this work are substantial both practically and theoretically. Practically, the approach offers a tool for enhancing LLM performance on real-world tasks that involve long-text comprehension. Theoretically, it underscores the importance of maintaining semantic integrity above blindly following architectural constraints like fixed-length chunking.

Speculation on Future Work

The insights from this paper pave the way for further investigations into adaptive methods for long-context processing. Future work may explore integrating this dynamic approach with more advanced neural architectures or developing methods for even more efficient chunk selection to accommodate increasing demands for computational efficiency. Additionally, there could be a focus on exploring other domains and task types beyond reading comprehension, as well as extending this work to multilingual contexts, ensuring adaptability across different languages and scripts.

Overall, this paper provides a significant contribution to the field by proposing a method that not only enhances the comprehension capabilities of LLMs for long contexts but also sets a new direction for further exploration into dynamic text processing techniques in natural language understanding.