Late Chunking in Dense Retrieval

• Late chunking is a contextualized embedding technique that first encodes full documents before applying segmentation, preserving cross-chunk discourse.
• It improves dense retrieval performance by maintaining global context, which enhances semantic matching in in-corpus search tasks.
• The approach relies on long-context token-level representations and mean pooling, though it may struggle with fine-grained in-document retrieval.

Searching arXiv for papers on late chunking and related chunking strategies. Late chunking is a post-embedding, or contextualized, chunking paradigm for dense retrieval and retrieval-augmented generation in which a document is encoded before it is segmented into retrievable units. Instead of splitting a document into chunks and embedding each chunk independently, a long-context embedding model first produces token-level contextualized representations over the full document, or over the largest window that fits its context length, and chunk vectors are then formed by pooling the token embeddings that fall inside later-applied segment boundaries. The method was introduced as a way to preserve cross-chunk context without abandoning chunk-level retrieval granularity, and subsequent work has situated it as an embedding-ordering strategy rather than a segmentation method in its own right (Günther et al., 2024, Zhou et al., 19 Feb 2026).

1. Definition and conceptual basis

Late chunking arose from a tension that is central to dense retrieval. On one hand, shorter retrieval units are often preferable because a single embedding vector can over-compress the semantics of long passages, and downstream LLMs are less efficient when supplied with long retrieved contexts. On the other hand, pre-segmenting a document and embedding each chunk independently removes surrounding context that may be necessary to interpret pronouns, ellipsis, discourse links, and entity references (Günther et al., 2024).

The defining inversion is therefore straightforward. In a conventional chunk-then-embed pipeline, a document is first partitioned and each chunk is embedded in isolation. In late chunking, the document is first processed by a long-context encoder, producing contextualized token embeddings, and chunk boundaries are applied only afterward. Mean pooling over the token span of each resulting segment yields the chunk embedding. In the taxonomy of later comparative work, this is the post-embedding side of the embedding-chunking ordering axis, also called contextualized chunking (Zhou et al., 19 Feb 2026).

The original motivation is especially clear in context-dependent passages. In the Berlin example used in the original paper, chunks such as “Its more than 3.85 million inhabitants...” and “The city is also one of the states of Germany...” do not explicitly mention “Berlin,” yet late chunking markedly increases their cosine similarity to the query “Berlin”: from 0.7084 to 0.8249 for the first chunk, and from 0.7535 to 0.8498 for the second. The mechanism is that the token representations for “its” and “the city” are computed after the model has already processed the full article context (Günther et al., 2024).

2. Position in chunking taxonomies

Later evaluations formalized chunking design space along two orthogonal dimensions: segmentation method and embedding-chunking ordering. Segmentation methods include structure-based procedures such as paragraph, fixed-size, and sentence chunking, as well as semantically informed or LLM-guided methods such as semantic splitting, proposition decomposition, and LumberChunker. Late chunking belongs exclusively to the second dimension. It does not determine where boundaries are; it determines whether chunk embeddings are computed from isolated chunk inputs or from token embeddings contextualized by a broader document span (Zhou et al., 19 Feb 2026).

This orthogonality is important because late chunking can be paired with a wide range of boundary schemes. The reproduced taxonomy explicitly treats paragraph splitting as newline-based segmentation with empty chunks removed, fixed-size chunking as constant token-length partitions, sentence chunking as grouped sentences, semantic chunking as similarity-thresholded sentence segmentation, proposition chunking as LLM decomposition into atomic facts, and LumberChunker as LLM identification of topic-shift breakpoints. Any of these can, in principle, be applied after contextual encoding rather than before it (Zhou et al., 19 Feb 2026).

A later hierarchical chunking study characterized one limitation of standard late chunking formulations in precisely these terms: late chunking preserves global context by embedding entire documents before chunking, but it still yields a flat chunk inventory rather than an explicitly hierarchical structure. That observation does not negate late chunking; it identifies a separate axis—hierarchical organization of chunks—that the basic late-chunking formulation does not address (Lu et al., 15 Sep 2025).

3. Algorithmic formulation and implementation

The core late-chunking pipeline is consistent across the primary text-retrieval papers. A document $T$ is tokenized into a sequence $(\tau_1,\dots,\tau_m)$, the full token sequence is passed once through the embedding model, and contextualized token representations are produced:

$$(\vartheta_1, \dots, \vartheta_m) \leftarrow \mathrm{Model}(\tau_1, \dots, \tau_m)$$

Chunk boundaries are then aligned to token spans, and each chunk embedding is mean-pooled from the token embeddings inside that span. In the original exposition, this is the essential novelty: chunk boundaries are used only after token-level contextualization (Günther et al., 2024).

A convenient equivalent statement, used in later RAG-oriented evaluation, is to let a document be $D=(t_1,\dots,t_n)$, encode the full document as $H=E(D)=(h_1,\dots,h_n)$, define a chunk $C_j=(t_{a_j},\dots,t_{b_j})$, and then compute

$e_j = \frac{1}{b_j-a_j+1}\sum_{i=a_j}^{b_j} h_i$

followed by vector normalization when cosine similarity retrieval is used (Merola et al., 28 Apr 2025).

Several implementation details recur. First, late chunking requires access to token-level representations before final pooling, which constrains model choice. Comparative evaluation over contextualized chunking used jina-embeddings-v2-small-en, jina-embeddings-v3, nomic-embed-text-v1, and multilingual-e5-large, with model selection explicitly limited to retrieval-capable encoders that employ mean pooling and expose token-level states (Zhou et al., 19 Feb 2026). Second, the method depends on long-context support. One evaluation describes feeding the maximum sequence that fits the model’s context window, with 8,192 tokens used as the motivating example, and partitioning longer documents into the minimum necessary number of windows. That same study does not specify overlap between such windows and does not specify how chunks crossing context-window boundaries are handled, so those behaviors remain unspecified there (Zhou et al., 19 Feb 2026).

The original paper also described boundary-alignment details that matter for faithful reproduction. Because chunkers may operate on character spans, token offsets are accumulated to translate character-defined chunks into token boundary cues. Special tokens inserted by the tokenizer, and prepended instruction text used by some models such as jina-embeddings-v3 and nomic-embed-text-v1, are not discarded wholesale; prepended-token embeddings are included in the first chunk’s mean pooling and appended-token embeddings in the last chunk’s mean pooling (Günther et al., 2024).

Late chunking was presented from the outset as an inference-time modification rather than a mandatory retraining regime, but the original work also proposed optional span-pooling fine-tuning. There, training data take the form $(q,d,\langle start,end\rangle)$, queries are aligned with annotated spans rather than whole documents, and the objective is an InfoNCE-based bidirectional pair loss. Empirically, those span-pooling variants yielded small but consistent improvements over mean-pooling-based training (Günther et al., 2024).

4. Empirical behavior across retrieval tasks

The first systematic evidence for late chunking came from comparisons against naive pre-chunking on four BeIR datasets—SciFact, NFCorpus, FiQA, and TRECCOVID—using three long-context embedding models. Averaged over 3 models × 4 datasets, the reported gains were +3.63% relative and +1.9 absolute nDCG@10 for sentence boundaries, +3.46% relative and +1.8 absolute for fixed-size boundaries, and +2.70% relative and +1.5 absolute for semantic sentence boundaries. The same study also showed that gains are especially strong for small chunks, while synthetic “needle in irrelevant context” tasks such as Needle-8192 and Passkey-8192 do not benefit because the surrounding context is “totally irrelevant” (Günther et al., 2024).

A broader 2026 reproduction and taxonomy paper substantially refined that picture by separating in-corpus retrieval from in-document retrieval. In-corpus retrieval, evaluated on six BEIR datasets with MaxP aggregation to document scores, generally benefited from contextualized chunking. Across models, the largest average relative gains appeared for proposition-based segmentation, ranging from +15.62% to +27.11%, while LumberChunker also improved consistently, from +2.42% to +4.80%. Structure-based methods improved more modestly, and semantic splitting was the main inconsistent case (Zhou et al., 19 Feb 2026).

The same evaluation found the opposite pattern for in-document retrieval on GutenQA, the paper’s “needle-in-a-haystack” setting. Contextualized chunking degraded performance across all tested configurations, with average relative changes of -5.16% for Jina-v2, -29.82% for Jina-v3, -53.17% for Nomic, and -3.83% for E5-large. The paper’s interpretation is task-specific: broader document context can help determine which document is relevant in a corpus, but within-document localization requires high discriminability among neighboring chunks, and contextualization can make chunks from the same document more similar by encoding broader document themes rather than chunk-specific details (Zhou et al., 19 Feb 2026).

The same study also separated context effects from chunk-size effects. Under pre-embedding chunking with Jina-v3, in-document effectiveness correlated moderately with chunk size—$r=0.57$ for paragraph, $r=0.52$ for sentence, and $(\tau_1,\dots,\tau_m)$0 for proposition segmentation—whereas in-corpus correlations were weak, roughly $(\tau_1,\dots,\tau_m)$1 to $(\tau_1,\dots,\tau_m)$2. Under contextualized chunking, the in-document size correlations weakened, and in-corpus correlations between chunk size and relative improvement were near zero, $(\tau_1,\dots,\tau_m)$3 to $(\tau_1,\dots,\tau_m)$4. The authors therefore concluded that late-chunking gains are not simply a chunk-size artifact (Zhou et al., 19 Feb 2026).

5. Comparisons, variants, and later developments

Late chunking has frequently been discussed alongside contextual retrieval, but the two are operationally distinct. In one comparative RAG study, late chunking preserved context through the embedding computation itself, whereas contextual retrieval used LLM-generated chunk summaries, hybrid dense-plus-BM25 retrieval, weighted rank fusion, and cross-encoder reranking. On a 20% NFCorpus subset with Jina-V3 and fixed windows, contextual retrieval outperformed late chunking across all reported metrics, with NDCG@10 0.308 vs 0.294, MAP@10 0.166 vs 0.160, and F1@10 0.202 vs 0.192. The same paper concluded that contextual retrieval preserves semantic coherence more effectively but requires greater computational resources, while late chunking is more efficient but tends to sacrifice relevance and completeness (Merola et al., 28 Apr 2025).

That comparative study also underscored model dependence. Late chunking was competitive or slightly better than early chunking for some model–dataset pairs, such as Jina-V3 and Stella-V5 on NFCorpus, but it failed badly for BGE-M3 on the same dataset and was clearly worse than early chunking for Stella-V5 on MSMarco. The authors therefore rejected any universal claim of superiority and presented late chunking as a training-free contextualization strategy that must be benchmarked for the target corpus and model family (Merola et al., 28 Apr 2025).

More recent work has extended late chunking rather than merely evaluating it. In biomedical full-text retrieval, GraLC-RAG combined late chunking with structure-aware boundary detection, token-level UMLS knowledge-graph infusion via SapBERT and a GAT, and graph-guided hybrid retrieval. That work found that content-similarity methods still achieved the highest MRR, reaching 0.517 on full text, whereas the strength of structure-aware late chunking lay in retrieval breadth: structure-aware methods retrieved from as many as 15.57 sections at top-20, compared with 1.00 for content-only methods, and KG-infused retrieval narrowed the downstream answer-quality gap to $(\tau_1,\dots,\tau_m)$5F1 = 0.009 while maintaining 4.6× section diversity (Mortezaagha et al., 23 Mar 2026).

The principle has also been generalized beyond text. In visual document retrieval, ColChunk formulated “Visual Late Chunking” as clustering contextualized patch embeddings after the LLM backbone rather than chunking images or OCR regions before encoding. Across 24 VDR datasets, the paper reported over a 90% reduction in storage requirements and a 9-point average improvement in nDCG@5 across representative single-vector models, showing that the contextualize-first, compress-later pattern can be reformulated in multimodal settings (Yan et al., 11 Apr 2026).

6. Limitations, misconceptions, and practical use

A recurring misconception is to treat late chunking as a boundary-finding algorithm. It is not. In the most systematic taxonomy, late chunking is an embedding-ordering strategy that can be paired with paragraph, fixed-size, sentence, semantic, proposition, or LumberChunker boundaries. The segmentation method decides where the boundaries are; late chunking decides whether the embeddings for those chunks are contextualized before those boundaries are applied (Zhou et al., 19 Feb 2026).

Its practical constraints are equally clear. The method requires token-level access before pooling, works most naturally with mean-pooling encoders, and depends on substantial context windows. That dependence is visible in mixed-model results: Jina-v2, Jina-v3, and Nomic support long inputs such as 8,192 tokens, whereas E5-large has a 512-token context limit, which constrains how much extra context contextualized chunking can exploit (Zhou et al., 19 Feb 2026). The original paper also noted that encoding long texts can become memory-intensive, describing memory growth during encoding as increasing “exponentially” with token count and motivating its long-late-chunking macro-chunk extension (Günther et al., 2024).

Failure modes are well documented. Beyond the consistent degradation in in-document retrieval, contextualized chunking plus semantic splitting degraded strongly on FiQA in the large 2026 evaluation, with relative drops of -26.37% for Jina-v2, -26.73% for Jina-v3, -25.43% for Nomic, and -23.56% for E5-large (Zhou et al., 19 Feb 2026). Separate RAG-oriented comparison showed dramatic failure for BGE-M3 on NFCorpus and clear underperformance on MSMarco for Stella-V5, again emphasizing that late chunking does not consistently outperform early chunking across models and datasets (Merola et al., 28 Apr 2025).

The implementation literature also leaves certain systems questions unresolved. Some evaluation papers quantify segmentation throughput—for example, LumberChunker: 1.11 docs/s versus paragraph-based: 1,854 docs/s—but do not provide equivalent end-to-end latency or memory benchmarks specifically for late chunking itself (Zhou et al., 19 Feb 2026). Likewise, when documents exceed the model context limit, some papers specify maximal windows and minimum necessary partitioning, but not overlap or cross-window stitching; others introduce macro-chunk overlap but do not fix a universal numeric policy (Günther et al., 2024, Zhou et al., 19 Feb 2026).

Within those constraints, a stable practical consensus has emerged. Late chunking is most useful for in-corpus retrieval, especially when chunk meaning depends strongly on surrounding discourse and when long-context embedding models are available. It is not a universal replacement for pre-embedding chunking, and it is ill-suited to in-document retrieval tasks where the central requirement is to discriminate among chunks from the same document. Later work suggests that its strongest future role may be as one component in richer systems—paired with hierarchical chunk structures, structure-aware boundary signals, or graph-guided retrieval—rather than as a standalone answer to the chunking problem (Zhou et al., 19 Feb 2026, Lu et al., 15 Sep 2025).