Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation

Abstract: Dense retrieval ranks one query vector against one document vector. On long documents, this interface can fail when a short but decisive span is weakened during document encoding before ranking. We study this failure mode as document-side early compression and introduce the Evidence Dilution Index (EDI) to measure how far a document-level representation falls below the strongest chunk-level evidence within the same gold document. Guided by this view, we propose DICE (Document Inference via Chunk Evidence), a training-free document-side strategy that splits documents into chunks, encodes them independently with a frozen model, and aggregates them back into a single vector while preserving the standard one-query-one-document interface. On LongEmbed, DICE improves retrieval across four backbones, with the largest gains on slices beyond 4k tokens: for Dream, Passkey >4k rises from 30.0 to 90.0 and Needle >4k from 23.3 to 74.0. Across 12,779 filtered samples, DICE yields lower EDI than the single-vector baseline in 92.8% of cases. These results establish document-level encoding as a practical and underexplored lever for long-document retrieval.

• The paper introduces DICE, a training-free method that mitigates representational compression by encoding document chunks separately and aggregating them.
• It presents the Evidence Dilution Index (EDI) to quantify how much localized evidence is lost when compressing lengthy documents into a single vector.
• Empirical results across multiple benchmarks show that chunk-wise encoding significantly improves recall and ranking accuracy in long-document retrieval tasks.

Improving Long-Document Dense Retrieval via Chunk Evidence Aggregation

Motivation and Failure Modes

Dense retrieval frameworks standardly operate by mapping a query and a document to a pair of fixed-size vectors, followed by similarity-based ranking. While this paradigm scales efficiently and is widely deployed, it profoundly underperforms on retrieval tasks where decisive evidence is embedded within lengthy documents—such as legal filings, meeting transcripts, and narrative works—especially when the crucial evidence is localized to brief spans amid extensive irrelevant or off-topic content. The core bottleneck arises from "document-side early compression": encoding the entire document into a single vector, thereby diluting salient evidence before any query-document comparison is performed.

Figure 1: A motivating long-document retrieval case illustrating evidence dilution when single-vector encoding under-ranks gold documents containing the answer.

Traditional long-context encoders supporting window sizes up to 32k tokens still exhibit marked retrieval quality degradation on long documents (2606.18781). The key issue is not context-window capacity, but rather representational compression: relevance localized to a brief span gets averaged with non-relevant content and loses discriminative power during encoding. This dilution renders dense retrieval brittle for applications requiring high recall on sparse evidence within lengthy texts.

Evidence Dilution Index and Analytical Framework

Addressing early compression necessitates a principled diagnostic for quantifying lost evidence. The paper introduces the Evidence Dilution Index (EDI), which formalizes the extent to which a global document embedding falls below the strongest chunk-level evidence within the gold document. EDI takes into account both the maximum query-chunk similarity and the average similarity, providing a normalized metric robust to context length and evidence concentration.

• Evidence Concentration (EC) measures the sharpness of local relevance (i.e., whether a few chunks dominate).
• EDI measures the normalized gap between the max chunk similarity and the global document similarity, facilitating granular error analysis and mechanism-driven retrieval system design.

This analytical lens transforms evidence dilution from intuition to actionable metric, opening new optimization targets for document-level encoding strategies.

DICE: Document Inference via Chunk Evidence

The paper proposes DICE, a training-free document-side remedy for evidence dilution. DICE modifies only the document encoding path: each document is split into chunks (by token count), chunks are encoded independently using local positional indices (reset per chunk), and chunk embeddings are aggregated to construct a single document vector for retrieval. The query encoding remains unaltered, preserving the canonical one-query-one-document interface and guaranteeing deployment compatibility.

Figure 2: Overview of DICE, which encodes document chunks independently and aggregates them, leaving the query path and retrieval interface unchanged.

Aggregation is query-independent and typically implemented via mean pooling, although max and top-k pooling (based on embedding norm) are explored. Overlapping and granularity of chunks are ablated for their effects on retrieval efficacy and computational efficiency.

Empirical Results on LongEmbed

Extensive evaluation is conducted on LongEmbed, comprising synthetic (Passkey, Needle) and real-world retrieval benchmarks (NarrativeQA, QMSum, SummScreenFD, WikiMQA). Backbones tested span both diffusion and autoregressive architectures (Dream, Llama3, Mistral, Qwen2).

Main Retrieval Results

DICE produces substantial gains, particularly on long-context slices ($>$4k tokens):

• For Dream, Passkey $>$4k improves from 30.0 to 90.0, and Needle $>$4k from 23.3 to 74.0.
• DICE outperforms single-vector baselines across backbones, especially in cases of localized evidence and lengthy documents.
• Ablation studies show that chunk size is critical; larger chunks (up to 1024 tokens) achieve higher hard-slice means.

  Figure 3: Needle Hit@1 (\%) by context length on Dream. DICE prevents sharp degradation in retrieval quality as context length increases.

  

  Figure 4: Dream chunk-size ablation indicating consistently greater retrieval accuracy for larger chunks.

  

  Figure 5: Quality-cost trade-off highlighting DICE's superior recall at increased document-side computational expense.

Mechanism Validation via EDI

Mechanistic validation uses EDI to demonstrate that DICE achieves lower evidence dilution across nearly all samples (median EDI: -0.709 for DICE vs. 0.055 for Single; DICE yields lower EDI than Single in 92.8% of robust cases). This holds for both synthetic and real data, and is consistent across backbone types.

Figure 6: Median EDI across EC terciles, showing DICE maintains lower dilution even as evidence localization increases.

Robustness and Deployment Considerations

• Chunk overlap incurs substantial computational cost without consistent accuracy gains; non-overlapping chunks are optimal for efficiency.
• Aggregation strategy: Mean pooling is more robust than max or top-k pooling; the latter degrade real-task performance.
• Encoding cost: DICE increases document-side computational load (3–4$\times$), but indexing operations are typically offline.

Synthetic Top-$k$ Performance and Transferability

DICE's retrieval advantage persists in top-$k$ ranking metrics and remains pronounced in longer contexts.

Figure 7: Per-length Hit@5 (\%) on synthetic tasks, showing DICE's benefit persists with increasing context size.

Figure 8: Needle Hit@k (\%) by length, indicating DICE's advantage as document length increases.

Figure 9: Passkey Hit@k (\%) by length, validating preservation of relevant evidence via chunk aggregation.

Further transfer experiments with FollowIR benchmarks confirm DICE's generalization potential, although optimal chunk configuration is task-dependent. Adjacent design references (e.g., LateChunk-Doc) indicate competitive results but reaffirm the necessity of local context preservation via chunking.

Theoretical and Practical Implications

This research reframes document-side encoding as an independent optimization axis for dense retrieval, distinct from the encoder architecture and query-side strategies. The findings empirically illustrate that:

• Evidence dilution is a core representational limitation, not merely a model scaling artifact.
• Delaying compression via independent chunk encoding, followed by conservative aggregation, robustly improves ranking for challenging long-document scenarios.
• EDI is a viable diagnostic for mechanism-driven retrieval improvement, applicable across domains.

Practically, DICE's compatibility with existing retrieval systems (no need to change the interface or retrain encoders) enables immediate application in downstream information retrieval pipelines, albeit at a moderate computational overhead. The theoretical implications point to future work on query-aware chunk aggregation, learned compression methods, and hybrid passage-multi-vector retrieval architectures.

Conclusion

This paper provides a formal treatment of evidence dilution in dense retrieval and introduces the DICE method for mitigating representational bottlenecks in long-document ranking. The proposed chunk-wise encoding and aggregation strategy yields marked improvements in recall for documents exceeding 4k tokens and enables granular mechanism validation via EDI. The results establish document-side encoding as a crucial, under-explored lever for retrieval improvement and invite further exploration of learned and query-dependent aggregation approaches to address the remaining efficiency-accuracy trade-offs.