LGMGC: Logits-Guided Multi-Granular Chunker

• The framework integrates LLM [EOS] logits with hierarchical chunking to optimize passage segmentation for improved retrieval accuracy.
• LGMGC is designed for document segmentation in RAG pipelines, enabling both precise and comprehensive retrieval of relevant passages.
• Empirical evaluations demonstrate that LGMGC achieves higher DCG and F1 scores, indicating robust retrieval performance and efficient QA synthesis.

The Logits-Guided Multi-Granular Chunker (LGMGC) is a framework designed for document segmentation in Retrieval-Augmented Generation (RAG) pipelines, specifically addressing optimal passage segmentation for open-domain question answering (QA). LGMGC utilizes LLM generation preferences, as reflected in end-of-sequence ([EOS]) token logits, to determine semantically coherent chunk boundaries and incorporates multi-granular segmentation to align retrieval granularity with query specificity. By integrating LLM-informed breakpoints with hierarchical chunking, LGMGC enhances the relevance and efficiency of both dense passage retrieval and end-to-end QA synthesis (Liu et al., 17 Jan 2025).

1. Motivation and Challenges in Document Chunking

In dense passage retrieval (DPR) and RAG frameworks, the document $D$ must be pre-segmented into a collection of chunks $C = \{c_1, \ldots, c_M\}$ prior to embedding and indexing. The retrieved chunks serve as constrained context for downstream LLM-based answer synthesis, making the chunking phase a key determinant of overall pipeline efficacy. Existing approaches exhibit notable deficiencies:

• Fixed-size chunkers: Sliding windows with fixed token count $\theta$ can bisect sentences or paragraphs, leading to scattered supporting evidence and impaired passage integrity.
• Rule-based or recursive chunkers: These rely on syntactic boundaries (e.g., headings, punctuation) but lack adaptation to where an LLM prefers to conclude semantically, often misaligning with the LLM’s generation behavior.
• Embedding-based chunkers: Semantic distance heuristics disregard LLM generative signals, neglecting implicit preferences revealed during generation.

These shortcomings propagate error, manifesting as increased retrieval noise (irrelevant false positives) and synthesis noise (excess, off-topic context), ultimately degrading QA precision and recall. LGMGC aims to optimize chunk formation by aligning boundaries with high [EOS] logit positions and by generating hierarchical (multi-granular) chunk sets tailored to varying query scopes.

2. Formalism and Core Modules

LGMGC consists of two tightly integrated modules: the Logits-Guided Chunker (LG) and the Multi-Granular Chunker (MG).

2.1 Notation and Problem Setup

• Let $D = [t_1, t_2, ..., t_N]$ be a tokenized document.
• Sentence segmentation yields $S = \{s_1, ..., s_n\}$, where $s_j = [t_{p_j}, ..., t_{p_{j+1}-1}]$.
• A chunk $c$ is any contiguous subsequence of sentences: $c = s_a \oplus s_{a+1} \oplus ... \oplus s_b$.
• The LLM provides a logit $\ell_i$ for the [EOS] token at every sentence end $i$. The [EOS] boundary score is $C = \{c_1, \ldots, c_M\}$0, the softmax probability of [EOS] given a prefix prompt $C = \{c_1, \ldots, c_M\}$1.

2.2 Logits-Guided Chunker (LG)

LG identifies optimal chunk boundaries by interpreting the LLM’s local [EOS] probability as a proxy for semantic completeness. For the current buffer of up to $C = \{c_1, \ldots, c_M\}$2 tokens, LG computes $C = \{c_1, \ldots, c_M\}$3 for every sentence boundary, then selects

$C = \{c_1, \ldots, c_M\}$4

The buffer up to $C = \{c_1, \ldots, c_M\}$5 forms the parent chunk $C = \{c_1, \ldots, c_M\}$6. The remainder is concatenated with the next $C = \{c_1, \ldots, c_M\}$7-token slice, and the process repeats until the whole document is segmented or falls below a length threshold.

2.3 Multi-Granular Chunker (MG)

MG recursively sub-divides each LG parent chunk $C = \{c_1, \ldots, c_M\}$8 into smaller “child” chunks at fractional window sizes, e.g., $C = \{c_1, \ldots, c_M\}$9 and $\theta$0, using the same sentence boundary constraints:

• $\theta$1,
• $\theta$2,
• $\theta$3.

Each child chunk is indexed. Upon query $\theta$4, the retriever returns the top-$\theta$5 child chunks (by similarity), which are regrouped by parent. The parent’s overall score is the maximum similarity over its children:

$\theta$6

with $\theta$7, $\theta$8 as dense embeddings. The top-$\theta$9 parent chunks (or their best child) are concatenated for LLM synthesis, up to the context window budget.

3. Unified LGMGC Pipeline

The end-to-end LGMGC segmentation and retrieval workflow encompasses the following:

1. Logits-Guided segmentation: Apply LG to obtain parent chunks $D = [t_1, t_2, ..., t_N]$0 from $D = [t_1, t_2, ..., t_N]$1.
2. Multi-granular expansion: For each $D = [t_1, t_2, ..., t_N]$2, generate child chunks at multiple granularities ($D = [t_1, t_2, ..., t_N]$3, $D = [t_1, t_2, ..., t_N]$4, $D = [t_1, t_2, ..., t_N]$5).
3. Embedding and indexing: All child chunks are embedded (e.g., BGE-Large, E5-Large) and stored in a dense index (e.g., FAISS).
4. Retrieval: Given query $D = [t_1, t_2, ..., t_N]$6, select top $D = [t_1, t_2, ..., t_N]$7 children, compute score for each parent, select top $D = [t_1, t_2, ..., t_N]$8 parents.
5. LLM synthesis: Concatenate the full parent(s) or their top-scoring children, not exceeding the LLM’s token budget (e.g., 1,500 words), and pass to the LLM for answer synthesis.

No architectural modifications to the DPR retriever or LLM are required; LGMGC exclusively restructures the chunking and indexing pipeline.

4. Empirical Evaluation

The LGMGC framework has been evaluated on both passage retrieval and end-to-end QA.

4.1 Datasets and Metrics

• Passage retrieval: GutenQA (narrative books, “needle in a haystack” QA). Metrics: DCG@k, Recall@k.
• QA tasks: LongBench (NarrativeQA, QasperQA, MultifieldQA). Metric: token-level F1.

4.2 Comparative Baselines

Methods compared include:

Method	Type	Key Strategy
Recursive Chunker	Rule-based	Punctuation hierarchy
Semantic Chunker	Embedding-based	Semantic distance
Paragraph-Level Chunker	Fixed	Paragraph splits
LumberChunker	LLM iterative	Generation-based
MG only	Ablation	Multi-granular only
LG only	Ablation	Logits-guided only
LGMGC	Hybrid	LG + MG

4.3 Passage Retrieval Performance

With a BGE-Large retriever and mean DCG@k over $D = [t_1, t_2, ..., t_N]$9:

• Recursive: DCG@1 = 47.2 ± 2.5
• Semantic: 41.3 ± 2.4
• Paragraph: 50.0
• LumberChunker: 55.7
• LG: 52.6 ± 2.3
• MG: 60.3 ± 3.8
• LGMGC: 63.0 ± 1.4 (best)

Recall@5 achieves 86.8% with LGMGC vs. 81.7% for LumberChunker.

4.4 End-to-End QA Results

F1 scores (best $S = \{s_1, ..., s_n\}$0) with BGE-Large retrieval and Llama3-8b synth:

Method	NarrativeQA	MultifieldQA	QasperQA	Avg
Recursive	18.1	49.3	39.1	35.5
Semantic	19.0	42.7	39.6	33.8
LGMGC	19.6	50.1	43.5	37.7

Results generalize across larger LLM synthesis models (Llama3-70b) and alternative dense retrievers (E5-Large).

5. Analytical Findings and Qualitative Observations

Ablation analysis indicates that the hybrid strategy (LG + MG; i.e., LGMGC) outperforms either component applied in isolation, confirming the complementarity of generative segmentation and multi-granular coverage. LGMGC exhibits superior robustness to base chunk size $S = \{s_1, ..., s_n\}$1, with the lowest performance variance, thereby reducing the burden of hyperparameter tuning. Empirically, greedy selection of the highest $S = \{s_1, ..., s_n\}$2[EOS]$S = \{s_1, ..., s_n\}$3 per buffer rather than naïve thresholding yields the best semantic chunk isolation. Qualitatively, dynamic chunk endpoints correlate with narrative “beats,” avoiding mid-sentence fragmentation observed under fixed-size chunking. Multi-granular variants also enable both pinpoint retrieval for narrow queries and verbose context for broad queries.

6. Limitations and Prospective Extensions

LGMGC incurs increased compute from one LLM forward pass per $S = \{s_1, ..., s_n\}$4-sized window; this is tractable with small LLMs, but costlier for larger models. The approach currently exploits only [EOS] logits; a plausible implication is that integrating alternate generative uncertainty signals (e.g., masked-LM surprisal, attention-based salience) could further optimize chunking quality. The use of static subdivisions ($S = \{s_1, ..., s_n\}$5, $S = \{s_1, ..., s_n\}$6) could be enhanced by making chunk partitioning dynamic and trainable, supporting end-to-end optimization for retrieval performance.

In summary, the Logits-Guided Multi-Granular Chunker leverages LLM generative behavior and multi-scale chunking to achieve state-of-the-art performance in dense retrieval and extractive QA, while offering architectural compatibility and operational efficiency (Liu et al., 17 Jan 2025).