MLNeedle: Benchmarking Long-Context Retrieval

• MLNeedle Benchmark is a collection of rigorous, needle-in-a-haystack evaluations designed to test long-context retrieval in LLMs and MLLMs across multimodal and multilingual settings.
• It leverages structured datasets with embedded needles amid thousands of distractors to assess attention, memory, and hallucination detection at extreme context lengths.
• Empirical results reveal performance drops in open-source models and highlight gaps in visual retrieval, cross-lingual alignment, and effective context size handling.

MLNeedle Benchmark refers to a family of "needle-in-a-haystack" benchmarks designed to rigorously evaluate the long-context information retrieval capabilities of LLMs and multimodal LLMs (MLLMs) in various settings—multimodal, multimodal document, and multilingual. Benchmarks under the MLNeedle paradigm assess a model’s ability to identify a relevant item ("needle"), such as a sub-image or text span, within a large noisy context ("haystack") that may contain hundreds of distractors. These benchmarks have become central to stress-testing attention, memory, and retrieval mechanisms under the demands of real-world document or image-scale inputs (Wang et al., 2024, Wang et al., 2024, Hengle et al., 2024).

1. Fundamental Objectives and Paradigm

The core objective of the MLNeedle benchmarks is to measure the robustness and precision of LLMs’ and MLLMs’ long-range retrieval when operating on input contexts that significantly exceed traditional evaluation regimes. By embedding a “needle” (a uniquely relevant passage, image patch, or text statement) within a large haystack, the benchmarks simulate realistic tasks such as cross-document search, multimodal report analysis, and multilingual information access.

This paradigm extends beyond short-context QA or image captioning: real-world applications require models to process and match information across extensive multimodal contexts, potentially comprising up to 640 sub-images (Wang et al., 2024), sequences of 72 K tokens (Wang et al., 2024), or multilingual texts spanning up to 32 K tokens (Hengle et al., 2024).

2. Design: Task Formulation and Dataset Construction

2.1. Multimodal Needle-in-a-Haystack (MMNeedle) (Wang et al., 2024)

The MMNeedle benchmark specifically targets MLLMs by constructing a retrieval task at the sub-image level:

• Haystack Construction: A sequence of $M$ "stitched images," each composed of $N \times N$ sub-images, resulting in contexts with up to $M \times N^2$ cells, where each sub-image is sourced from MS COCO and resized to $256 \times 256$.
• Needle Definition: The "needle" is a specific sub-image whose ground-truth caption uniquely describes its contents.
• Labeling Protocol: For each haystack, positive labels assign the exact (m, r, c) location of a needle; negative samples use captions of images not present in the haystack (label: –1). Generation is fully automatic via MS COCO annotations and random sampling, leading to 280,000 balanced positive/negative pairs.

2.2. Needle in a Multimodal Haystack (MM-NIAH) (Wang et al., 2024)

MM-NIAH targets comprehension over extremely long, interleaved image–text documents:

• Document Base: Uses OBELICS, a large-scale web-derived set of interleaved image–text pages, concatenated to contexts up to 72 K tokens with up to 36 images.
• Needle Types: Injects (i) manually written text needles or (ii) synthetic image needles (cartoon or sampled) into the document at diverse depths (early, middle, late).
• Annotation: Each task instance contains one type of needle and one evaluation task: retrieval, counting, or reasoning, for a total of approximately 12,000 samples across 6 evaluation modalities.

2.3. Multilingual Needle-in-a-Haystack (MLNeedle) (Hengle et al., 2024)

MLNeedle evaluates multilingual LLMs on long-context QA retrieval:

• Document Pool: Needles are QA passages from MLQA in seven languages; distractors are Wikipedia-based passages from mMARCO, selected for maximal semantic similarity but guaranteed not to contain the answer.
• Needle Language and Position: The needle may be in any of the seven languages, positioned at the beginning, middle, or end of the haystack to probe positional memory effects across varying context lengths (4 K–32 K tokens).

3. Evaluation Protocols and Metrics

All MLNeedle variants employ metrics designed to disentangle retrieval capacity, positional/attentional biases, and hallucination susceptibility.

3.1. MMNeedle (Wang et al., 2024)

• Existence Accuracy: Binary correctness of present/absent needle detection.
• Index Accuracy: Correct identification of the needle’s image index $m$.
• Exact Accuracy: Precise localization ((m, r, c)) across all needles; hierarchy: $$\text{ExistenceAcc} \geq \text{IndexAcc} \geq \text{ExactAcc}$$.

3.2. MM-NIAH (Wang et al., 2024)

• Retrieval (P@K): Fraction of top-K predictions matching ground truth.
• Counting (Soft Acc/RMSE): Overlap between predicted and true counts (vector) or root mean squared error.
• Reasoning (Acc): Proportion of correct answers (open/fixed choice).

3.3. MLNeedle (Hengle et al., 2024)

• Exact Accuracy: Fraction of queries for which any output string matches the gold answer span.
• Existence Accuracy: Yes/No correctness of whether an answer exists within the haystack.
• Monolingual vs. Cross-Lingual: Disaggregates performance when needle and haystack share or differ in language.

4. Empirical Findings and Insights

4.1. MMNeedle (Wang et al., 2024)

• Model Performance: GPT-4o achieves highest exact accuracy (≈97 % at moderate context, dropping sharply to ≈1 % at maximal context size), with Gemini 1.5 Pro second-best. Open-source models’ exact accuracy rapidly approaches zero beyond a few dozen sub-images.
• Hallucination: High rate of false positive detections on negative samples; e.g., GPT-4o at (10,4): existence accuracy ≈1 % → hallucination ≈99 %.
• API vs. Open-Source Gap: API-based models outperform, likely reflecting larger capacity and more extensive vision-language pretraining. Open-source models lack robust patch-level memory/fusion.

4.2. MM-NIAH (Wang et al., 2024)

• Context Degradation: All evaluated models (including GPT-4V and Gemini-1.5) display rapid performance drop beyond 16 K tokens, especially on visual retrieval and counting.
• Modality Gap: Vision-centric tasks are significantly harder; e.g., retrieval-image ≈20–30 % (InternVL) vs. retrieval-text ≈90 %. Best human performance remains ≈99 % across context length.
• RAG (Retrieval-Augmented Generation): Strong gains for text-needle tasks (P@K∼97 % with InternVL+RAG), but little improvement for image retrieval/counting due to non-differentiable visual retrieval.

4.3. MLNeedle (Hengle et al., 2024)

• Length Robustness: No evaluated model matches its claimed context window; “effective length” (75 % of baseline accuracy) is generally much smaller (e.g., Mistral ≈8 K vs. 32 K claimed).
• Lost-in-the-Middle: All models suffer substantial (10–20 pp) accuracy drop when the needle is placed in the middle of the input.
• Cross-Lingual Deficit: Retrieval accuracy is highest for English needles (Mistral: 0.68), moderate for German/Spanish, and much lower for Arabic/Chinese (0.24–0.31). Distractor language has minimal effect; primary degradation arises when the relevant passage is non-Latin or low-resource.

Benchmark	Context Length (max)	Visual/Text	Retrieval Metric	Key Deficiency Exposed
MMNeedle	up to 640 images	Visual	ExactAcc/IndexAcc	Capacity, hallucination in negative samples
MM-NIAH	72 K tokens/36 imgs	Multimodal	P@K/SoftAcc/Acc	Modality gap, context degradation
MLNeedle	32 K tokens	Text	ExactAcc/ExistAcc	Cross-lingual deficit, position bias

5. Architectures Evaluated

MLNeedle benchmarks test both API-based (proprietary) and open-source systems:

• API-Based (MMNeedle/MM-NIAH): GPT-4o, GPT-4V, Claude 3 Opus, Gemini 1.0 Pro, Gemini 1.5 Pro.
• Open-Source (MMNeedle/MM-NIAH): CogVLM (17B, CogVLM2-Llama-3), Fuyu-8B, mPLUG-Owl-v2, InstructBLIP (Vicuna-13B, Flan-T5-XXL), IDEFICS2-8B, LLaVA-Llama-3, Emu2-Chat, VILA-13B, InternVL-1.5.
• Multilingual (MLNeedle): Mistral-7B-Instruct-v0.2, Cohere Aya-23-8B, Llama3-8B-Instruct, Llama2-7B-Chat.

These benchmarks reveal significant performance stratification and underscore the pressing need for improved attention mechanisms, patch-level vision fusion, cross-lingual alignment, and positional information handling.

6. Community Impact and Future Directions

MLNeedle-style benchmarks have shaped research priorities toward:

• Long-Context Robustness: More realistic models of application-scale document/image context, going far beyond single-image QA or short passage retrieval.
• Hallucination Mitigation: Necessity for explicit negative-class training, memory-augmented architectures, and attention schemes that avoid rampant false positives in sparse-relevance scenarios.
• Multilingual and Vision-Language Alignment: Research into pretraining with ultra-long documents in multiple scripts, cross-modal memory architectures for image needles, and hierarchical/sparse positional encodings.
• Reproducibility: All benchmark code, datasets, and evaluation harnesses are released for community use, e.g. MMNeedle GitHub (Wang et al., 2024), MM-NIAH GitHub (Wang et al., 2024).

A plausible implication is that further advances in long-context capabilities will require not only architectural innovation but also dataset and metric development that directly targets retrieval in complex, heterogeneous, and multilingual input settings.

7. Related Benchmarks and Distinctions

The MLNeedle paradigm distinguishes itself from prior benchmarks by (a) systematically scaling context length, (b) integrating negative samples to expose hallucination, (c) spanning both unimodal, multimodal, and multilingual settings, and (d) explicitly disentangling modality and language-specific failure modes.

While MM-NIAH and MLNeedle share roots in the “needle-in-a-haystack” concept, MMNeedle (focused on fine-grained visual retrieval via image stitching) and MLNeedle (emphasizing multilingual passage retrieval) surface orthogonal dimensions of long-context reasoning. These benchmarks collectively constitute the current standard for evaluating and advancing model architectures capable of sustained, accurate retrieval in the face of vast and noisy real-world input environments (Wang et al., 2024, Wang et al., 2024, Hengle et al., 2024).