Document Haystack: Retrieval & Reasoning

• Document Haystack is a set of evaluation settings where models locate sparse, relevant evidence amid extensive distracting contexts.
• It encompasses tasks such as long-context retrieval, multi-hop reasoning, and multimodal interpretation across varied benchmarks and metrics.
• Key insights include addressing positional bias, modality challenges, and the need for robust retrieval beyond simply enlarging context windows.

Document Haystack denotes a family of evaluation settings in which a model must locate, aggregate, or reason over a small amount of relevant evidence embedded within a much larger distracting document context. In recent work, the paradigm has expanded from single-span retrieval in long textual contexts to multilingual retrieval, multi-hop reasoning with multiple supporting documents, query-focused summarization with source attribution, multimodal page-level search in long PDFs, and vision-language retrieval over hundreds to thousands of documents (Hengle et al., 2024, Wang, 5 Apr 2025, Laban et al., 2024, Huybrechts et al., 18 Jul 2025, Chen et al., 2024, Kanagalingam et al., 24 Jun 2026). Across these variants, the central question is not merely whether a model can ingest long input, but whether it can use the relevant evidence under realistic distraction, positional bias, modality mixing, and retrieval noise.

1. Conceptual definition and benchmark families

The minimal document-haystack abstraction is explicit in multilingual long-context retrieval: a question $Q$ is paired with $K$ passages, exactly one of which is the answer-bearing needle $N$, while the remaining $K-1$ passages form the haystack $H$; the model receives a linearized context $C$ and must answer using only the provided passages (Hengle et al., 2024). A more retrieval-system-oriented formalization appears in HaystackCraft, where the haystack is an ordered set of retrieved documents constructed under a target token budget $S$, optionally extended into multi-round agentic workflows in which the model refines its own queries and accumulates intermediate analyses (Li et al., 8 Oct 2025).

Across the literature, “Document Haystack” therefore names a family rather than a single benchmark. Some variants stress pure retrieval, some require multi-hop synthesis across multiple needles, some evaluate grounded summarization, and others foreground multimodal search in visually dense corpora.

Benchmark family	Haystack construction	Primary objective
MLNeedle	One answer-bearing passage plus multilingual distractors in 4K–32K context	Monolingual and cross-lingual long-context retrieval
MNIAH-R	Two supporting documents and eight distractors inserted into a long text haystack	Multi-hop retrieval and reasoning
SummHay	Roughly 100 documents with repeated subtopic-specific insights	Query-focused summarization with citations
Document Haystack	Financial reports of 5–200 pages with inserted text or text+image needles	Long-document VLM retrieval
MM-NIAH	Interleaved image-text sequences with text and image needles	Multimodal retrieval, counting, and reasoning
DocHaystack / InfoHaystack	Pools of 100–1000 visual documents with one relevant item	Large-scale visual document retrieval and QA
Invoice Haystack	1,500 visually homogeneous invoices	Retrieval under strong visual homogeneity

This family resemblance is methodological rather than superficial. All variants operationalize a high noise-to-signal regime, but they differ in what counts as signal: a single answer span, multiple evidence documents, repeated insights, a keyed visual cue, or the one correct document in a large repository (Wang, 5 Apr 2025, Laban et al., 2024, Huybrechts et al., 18 Jul 2025, Wang et al., 2024, Chen et al., 2024, Kanagalingam et al., 24 Jun 2026).

2. Textual long-context retrieval and multilingual document haystacks

MLNeedle provides the clearest textual document-haystack formulation for multilingual LLMs. It fixes the question language $\ell_q$ to English, varies the needle language $\ell_s$ and the haystack language across English, Arabic, German, Spanish, Hindi, Vietnamese, and Simplified Chinese, and tests context lengths at approximately 4K, 8K, 16K, and 32K tokens by increasing the number of distractors (Hengle et al., 2024). Distractors are drawn from mMARCO’s multilingual Wikipedia passages, embedded with paraphrase-multilingual-mpnet-base-v2 in a 768-dimensional multilingual sentence-BERT space, ranked by cosine similarity, and ordered by decreasing relevance in the final context.

The benchmark reports both exact accuracy and existence accuracy. Exact accuracy is the proportion of samples whose prediction contains any ground-truth answer; existence accuracy measures whether the model correctly indicates that an answer-bearing passage exists in the context. Because predictions may be produced in different languages, model outputs are translated to English via Google Translate API before comparison to the MLQA English golden answer, explicitly to reduce false negatives from language mismatches (Hengle et al., 2024).

The principal findings are restrictive. Claimed context windows are consistently larger than effective usable lengths. Averaged across seven languages, Llama2-7B-Chat has baseline 0.335, falls to 0.171 at 4K, and has effective length below 4K; Llama3-8B-Instruct has baseline 0.622, 0.479 at 4K, 0.295 at 8K, and effective length 4K; Cohere-Aya-23-8B has baseline 0.700, 0.460 at 4K, 0.449 at 8K, and effective length 4K; Mistral-7B-Instruct-v0.2 has baseline 0.579, 0.485 at 4K, 0.455 at 8K, 0.427 at 16K, 0.397 at 32K, and effective length 8K (Hengle et al., 2024). In this literature, “effective length” is operationalized as the maximum context length at which performance does not drop by more than 25% from baseline no-distractor accuracy.

Position and language are dominant factors. Accuracy is highest when the needle is at the beginning or end of the context and drops markedly in the middle, yielding the now-familiar U-shaped “lost-in-the-middle” profile. Needle language is far more consequential than haystack language: performance is strongest when the needle is in English or languages described as close to English, such as German and Spanish, and drops for non-Latin scripts such as Chinese and Arabic and for lower-resource Hindi. By contrast, changing the language of the non-relevant distractors has limited effect when the needle is fixed (Hengle et al., 2024). This establishes a recurring distinction in document-haystack studies: failures often arise from weak needle comprehension and retrieval rather than from distractor language alone.

3. From retrieval to reasoning, summarization, and agentic haystacks

The retrieval-only formulation was quickly judged insufficient for modern long-context models. MNIAH-R replaces the single answer-bearing span with two supporting documents required for multi-hop reasoning and inserts those two needles together with eight distractors into a long text derived from PaulGrahamEssays (Wang, 5 Apr 2025). The questions come from IRE and HotpotQA dev_distractor, and the paper foregrounds a crucial confounder: memory-based answering. To suppress direct answering from parametric knowledge, questions are retained only when a model answers correctly with the haystack and incorrectly without it, formalized as

$$\text{Acc}_M(Q\mid H\!\cup\!S\!\cup\!D)=1 \quad \text{and} \quad \text{Acc}_M(Q\mid \emptyset)=0.$$

After filtering, a clear accuracy decline appears with increasing context length for all evaluated models, and the paper argues that this decline is driven primarily by shrinking “thinking process length,” not by needle placement or inter-needle distance (Wang, 5 Apr 2025). The quantity

$K$0

is used to measure the length of explicitly generated reasoning, and accuracy tracks this thinking budget. The proposed mitigation is a decomposition into retrieval and reasoning plus a reflection mechanism: retrieve evidence, reason, reflect and retrieve again without repetition, then reason again over the accumulated evidence. Empirically, performance improves with more iterations and saturates after round 3. On Llama-3-8B-ProLong, the long-context degradation metric $K$1 falls from 25.8 in the baseline to 4.6 when the model is trained on iterative thinking traces (Wang, 5 Apr 2025).

SummHay shifts the task from answer extraction to grounded summarization. Each haystack contains roughly 100 documents and about 100k tokens, with repeated subtopic-specific “insights” distributed across the corpus; the system must answer a subtopic-derived query with a bullet-point summary and precise bracketed document citations (Laban et al., 2024). Evaluation is automatic because the benchmark controls both which insights should be covered and which documents contain them. Coverage averages per-insight labels in $K$2, Citation averages citation F1 over covered insights, and the Joint score multiplies coverage and citation contribution per insight:

$K$3

The benchmark is intentionally demanding. Human performance in the Oracle setting is about 56.1 Joint, while the best reported Oracle system reaches 44.6; full-context long-context LLMs without retrieval are much weaker, with GPT-4o at about 11.4 Joint and Claude 3 Opus at about 18.0 (Laban et al., 2024). SummHay also reproduces positional sensitivity: ordering relevant documents at the top or bottom of the context materially changes results.

HaystackCraft extends the paradigm again by insisting that realistic haystacks are retrieval-shaped and agentically evolving rather than synthetically static. It constructs haystacks from full English Wikipedia articles under sparse, dense, hybrid, and graph-reranked retrieval, then evaluates both static and dynamic settings in which the model may refine queries, reflect, or decide when to stop (Li et al., 8 Oct 2025). The main reported pattern is twofold: strong dense retrievers introduce more challenging semantic distractors, while graph-based Personalized PageRank reranking simultaneously improves retrieval effectiveness and mitigates more harmful distractors; meanwhile, multi-round agentic operation is brittle, with cascading failures and poor early stopping even for frontier models (Li et al., 8 Oct 2025). This suggests that document-haystack evaluation has moved from static recall probes toward end-to-end robustness tests for RAG and agentic search.

4. Multimodal long documents and page-level haystacks

The multimodal branch of the literature asks whether a model can find and use text or images scattered throughout a long visual document. MM-NIAH is the first benchmark explicitly designed for long multimodal document comprehension, building interleaved image-text haystacks from OBELICS and inserting either text needles or image needles (Wang et al., 2024). It evaluates three task types—retrieval, counting, and reasoning—each in both text-centric and image-centric form, over contexts from 1k to 72k image-plus-text tokens and up to 36 images. Counting uses a soft accuracy metric,

$K$4

where $K$5 is the number of matched elements and $K$6 is the ground-truth list length.

The benchmark’s results show a sharp asymmetry between text and vision. Overall performance is 42.4 for InternVL-1.5, 50.1 for InternVL-1.5-RAG, 55.3 for Gemini-1.5-Flash, and 98.9 for humans (Wang et al., 2024). Retrieval and reasoning over text needles are substantially easier than counting or retrieving image needles; RAG gives large gains on text tasks but little or no help on image tasks; and a lost-in-the-middle pattern appears for text needles. The authors also note that long-context capability from the underlying text-only LLM does not transfer cleanly to the multimodal model, and that training on interleaved image-text corpora is not by itself sufficient (Wang et al., 2024).

The benchmark named “Document Haystack” in vision-language evaluation moves from synthetic interleaving to native long PDFs and page renders. It uses 25 publicly available financial 10-K reports, trims each into 5, 10, 25, 50, 75, 100, 150, and 200 page variants, inserts either text needles or text+image needles at controlled depths, and releases PDFs, page images rendered at 200 DPI, and parsed text for the text-needle subset (Huybrechts et al., 18 Jul 2025). The scale is 400 document variants and 8,250 questions. Questions are templated as “What is the secret KEY in the document?”, and scoring is objective lowercased exact substring match, with alias lists for image needles (Huybrechts et al., 18 Jul 2025).

The empirical contrast between text-only and image-based access is pronounced. When text needles are retrieved from parsed document text, accuracy remains high and stable: at 200 pages, Nova Lite is 89.9 and Gemini Flash-2.0 is 91.8, while GPT-4o-mini reaches 98.4 up to 100 pages before token limits intervene (Huybrechts et al., 18 Jul 2025). The same text needles read from page images degrade much more strongly, and mixed text+image retrieval is hardest, falling to about 35–40% accuracy at 200 pages. The paper explicitly interprets this as evidence that much of the difficulty lies in reading text inside images and in cross-modal grounding rather than in long-context textual reasoning alone. It also documents a tokenization trade-off: Nova Lite uses about 1,626 tokens per image, Gemini Flash-2.0 about 258, and GPT-4o-mini about 636, which becomes consequential over hundreds of pages (Huybrechts et al., 18 Jul 2025).

5. Multi-document visual haystacks: heterogeneity, homogeneity, and verification

A separate line of work treats the haystack not as a single long document but as a pile of hundreds or thousands of separate visual documents. “Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents” introduces DocHaystack and InfoHaystack for this regime (Chen et al., 2024). DocHaystack is curated from DocVQA and InfoHaystack from InfographicVQA after a filtering pipeline that removes general questions and items answerable without vision. The final evaluation sets contain 109 questions supported by 59 evidence documents for DocHaystack and 155 questions supported by 66 evidence documents for InfoHaystack, each evaluated with haystack sizes of 100, 200, and 1000 documents (Chen et al., 2024).

The associated V-RAG system uses an ensemble of CLIP, SigLIP, and OpenCLIP encoders, averages their cosine-similarity scores, retains the top $K$7 candidates, filters them with a yes/no LLaVA-OneVision relevance check, and passes the final top $K$8 documents to a VQA model (Chen et al., 2024). On DocHaystack-1000, V-RAG reaches Recall@1 of 66.06, compared with 56.88 for BM25 and markedly lower scores for the single-encoder baselines; on InfoHaystack-1000 it reaches 64.52, compared with 53.55 for OpenCLIP (Chen et al., 2024). When integrated with generation, retrieval quality becomes the primary bottleneck: Qwen2-VL fine-tuned with distractors plus V-RAG attains 73.39 accuracy on DocHaystack-1000 and 60.00 on InfoHaystack-1000 (Chen et al., 2024).

Invoice Haystack pushes this regime into strong visual homogeneity. It contains 1,500 anonymized invoice images and 200 discriminative question-answer pairs, and defines the central failure mode as embedding collapse: visually homogeneous documents cluster so tightly in embedding space that similarity ranking cannot separate the correct invoice reliably (Kanagalingam et al., 24 Jun 2026). The benchmark quantifies this with mean pairwise cosine similarity of 0.73, versus 0.38 for DocHaystack and 0.31 for InfoHaystack (Kanagalingam et al., 24 Jun 2026). The proposed VL-RAG responds by combining OCR-derived text embeddings from BGE-Large with visual embeddings from SigLIP and OpenCLIP, averaging the three similarities, and then applying a VLM-based yes/no verification filter over the top-$K$9 shortlist.

The resulting gains are large precisely where homogeneity is strongest. VL-RAG reaches 60.0% Recall@1 on Invoice Haystack-500, compared with 46.5% for V-RAG; 53.0% on Invoice Haystack-1000, compared with 43.0%; and 50.0% on Invoice Haystack-1500, compared with 40.0% (Kanagalingam et al., 24 Jun 2026). It also improves DocHaystack-1000 from 75.2% to 77.1% and InfoHaystack-1000 from 80.0% to 84.5%. The paper’s interpretation is explicit: dual-modality fusion combats embedding collapse, while verification resolves residual ambiguity among near-duplicate templates (Kanagalingam et al., 24 Jun 2026). A plausible implication is that, in document haystacks, retrieval difficulty is not determined only by corpus size; visual homogeneity and answer discriminativeness can dominate.

6. Extensions, methodological lessons, and recurrent failure modes

Some recent work generalizes the haystack logic beyond document retrieval itself. “Semantic Needles in Document Haystacks” treats pairwise document similarity scoring as a needle-in-a-haystack problem by inserting a single semantically altered sentence into surrounding context and asking an LLM judge for a 0–100 similarity score (Aksoy et al., 20 Apr 2026). Across 5 LLMs, 3 perturbation types, 2 hay types, and 100 positional settings—3000 parameter settings in total—the study finds a within-document positional bias distinct from candidate-order effects: most models penalize earlier semantic differences more harshly, random surrounding context lowers scores and produces bipolarized distributions, and all models share the perturbation hierarchy

$N$0

meaning negation is penalized least and conjunction swap most (Aksoy et al., 20 Apr 2026). This extends document-haystack methodology from retrieval accuracy to judge sensitivity audits.

“Label-in-a-Haystack” applies a similar logic to subjective annotation correction. The query document and its candidate label are embedded among demonstrations, but the model is instructed to perform the task rather than copy the label; failure to copy is treated as informative signal rather than error (Chochlakis et al., 22 May 2025). In the LiaHR framework, if the model outputs $N$1, the generated label is used as a rectification. The paper reports that this improves downstream Demux performance slightly but consistently: on GoEmotions, Micro F1 rises from $N$2 with original labels to $N$3 with LiaHR Replace; on SemEval, from $N$4 to $N$5 (Chochlakis et al., 22 May 2025). Human evaluations further show preference for rectified labels when copy failure occurs. This suggests that “haystack” can denote a prompt configuration in which the relevant target is hidden amid demonstrations rather than documents.

Across the broader document-haystack literature, several failure modes recur. Advertised context windows exceed effective usable windows; relevance in the middle of a context is systematically harder to recover than relevance at the beginning or end; dense retrievers may improve recall while worsening distractor quality; reading text from images is far harder than reading the same content from parsed text; and visually homogeneous repositories expose embedding collapse that is largely invisible in heterogeneous benchmarks (Hengle et al., 2024, Li et al., 8 Oct 2025, Huybrechts et al., 18 Jul 2025, Kanagalingam et al., 24 Jun 2026). The practical recommendations likewise recur: vary needle position; report effective rather than claimed context length; include non-Latin and lower-resource languages; filter memory-based shortcuts; test retriever-ranked as well as randomized orderings; use verification stages in visually ambiguous corpora; and prefer evaluation protocols that expose real retrieval and reasoning bottlenecks rather than synthetic pass-through success (Hengle et al., 2024, Wang, 5 Apr 2025, Li et al., 8 Oct 2025).

A persistent misconception is that document-haystack performance is equivalent to raw context-window size. The surveyed benchmarks collectively argue otherwise. Usable long-context behavior depends on retrieval quality, position, modality, grounding strategy, and the model’s ability to preserve a sufficient reasoning budget as input grows. In that sense, document haystacks have become a general stress test for whether long-context systems can do more than accept long input: they test whether models can recover, combine, verify, and justify the one or few pieces of evidence that actually matter.