Finding Needles in Images (NiM) Benchmark

• Finding Needles in Images (NiM) is a benchmark that targets extracting and reasoning about tiny, spatially-localized evidence from complex document images.
• It constructs a diverse corpus using automated generation combined with human verification, ensuring detailed, patch-level question-answer pairs across various domains.
• The associated Spot-IT method enhances multimodal model performance by providing a zero-shot, training-free mechanism to focus on critical regions with adaptive Gaussian attention.

Searching arXiv for the NiM benchmark paper and closely related “needle in images” work to ground the article in current literature. Finding Needles in Images (NiM) is a benchmark and problem formulation for locating and reasoning about fine-grained details inside visually complex document images. In this setting, the decisive evidence is a spatially localized region that occupies less than $5\%$ of the total image area, so the task is not broad document understanding but the recovery of small, visually embedded clues within menus, newspapers, academic pages, magazines, website screenshots, and classroom lecture images (Thakkar et al., 7 Aug 2025). NiM therefore studies a document-centric “needle-in-haystack” regime in which multimodal models must both attend to the correct region and use that region for answer generation.

1. Definition and conceptual boundaries

NiM formalizes fine granularity as

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$

and uses this criterion to target tasks in which the relevant evidence is minute relative to the page (Thakkar et al., 7 Aug 2025). The motivating examples are document queries such as locating a low-cost breakfast extra on a restaurant menu, finding a disclaimer in a lengthy multi-column newspaper article, or identifying a parameter value or citation on an academic paper page. In each case, the required information is embedded inside a broader visual narrative rather than presented as a dominant object or a globally salient region.

NiM is adjacent to, but distinct from, several neighboring formulations. In Small Object Image Retrieval (SoIR), the goal is to retrieve images containing a specific small object in a cluttered scene while retaining a single compact descriptor per image; that literature emphasizes retrieval benchmarks such as INSTRE-XS, INSTRE-XXS, VoxDet-based benchmark, and PerMiR, rather than question answering over document pages (Green et al., 10 Mar 2025). VisualNeedle studies active visual search in information-dense scenes and evaluates tool-enabled cropping, no-tool reasoning, and crop-black ablations, with tasks spanning OCR, color recognition, entity recognition, spatial relation judgment, and occluded object recognition (Chen et al., 25 May 2026). Tiny-object classification work instead studies very low Object to Image (O2I) ratios in large images, asking whether a small target is present rather than extracting a localized textual or semantic answer from a document (Pawlowski et al., 2019). This suggests that NiM occupies an intermediate position between document VQA, fine-grained localized reasoning, and search-conditioned perception.

2. Corpus construction and annotation protocol

The NiM benchmark comprises six document domains and combines automatic generation with human verification. Documents were collected from real-world sources and rendered as images while preserving layout complexity. The benchmark contains 284 documents, about 2,970 images, and 1,180 question-answer pairs; average question length is about 11 tokens, and average answer length is about 2 tokens (Thakkar et al., 7 Aug 2025).

During annotation, each document image is divided into variable-sized grids from $2\times2$ to $6\times6$, and question-answer pairs are generated to focus on information localized within individual patches. The pipeline is hybrid: an initial automatic stage proposes questions, followed by human verification and additional manual creation, particularly for difficult domains. Every question-answer pair is reviewed by three independent annotators for accuracy, naturalness, and fine-grained focus (Thakkar et al., 7 Aug 2025).

Component	Instances
Document domains	Restaurant Menus; Academic Papers; Magazines; Newspapers; Website Screenshots; Classroom Lecture Screenshots
Question types	Inline; Boolean; Comparative; Complex Reasoning; Commonsense; Unanswerable
Verification	Three independent annotators per question-answer pair

The use of patch-level generation is central to the benchmark’s definition of difficulty. Rather than asking global questions about the page, NiM constructs queries anchored to local evidence and then checks that the questions remain natural and answerable. The data summary also reports that machine-generated questions are often indistinguishable from human ones in blind Turing tests, with Cohen’s $\kappa$ scores indicating moderate agreement (Thakkar et al., 7 Aug 2025).

3. Task structure and evaluation methodology

NiM evaluates multimodal models under standard document-QA metrics: Exact Match (EM), F1, and ANLS. The benchmark also includes qualitative assessment through Turing tests between human- and machine-generated questions, human response time versus accuracy, and robustness tests with distractor images (Thakkar et al., 7 Aug 2025).

The question taxonomy spans direct extraction and localized reasoning. Inline questions ask for direct detail extraction from a specific region. Boolean questions ask yes/no judgments about local specifics. Comparative questions compare nearby visual or textual elements. Complex Reasoning questions require multi-step reasoning within localized context. Commonsense questions require external but relevant world knowledge. Unanswerable questions test whether a model can recognize insufficient evidence (Thakkar et al., 7 Aug 2025). The benchmark therefore measures not only whether a model can read small details, but also whether it can combine local evidence with reasoning operations.

A related methodological issue appears in VisualNeedle, which explicitly tests whether success depends on genuine intermediate visual evidence. That benchmark includes text-only, no-tool, tool-enabled, and crop-black settings, where crop-black preserves the same tool-call trajectory but replaces returned crops with black images of the same size (Chen et al., 25 May 2026). A plausible implication is that NiM-style evaluation can benefit from the same concern: strong answer accuracy is not sufficient unless the model’s behavior is coupled to the correct local evidence.

4. Spot-IT: query-guided patch selection and Gaussian attention

The principal method proposed alongside NiM is Spot-IT, a zero-shot, training-free preprocessing pipeline that aims to guide an MLLM toward the correct fine-grained region (Thakkar et al., 7 Aug 2025). The image is subdivided into an $n\times n$ grid of patches, with $n=6$ reported as working well. A cleaned query $q_c$ and each patch $P_{ij}$ are embedded with SigLip: $$v_q = \text{SigLip}(q_c), \quad v_{ij} = \text{SigLip}(P_{ij})$$ and their cosine similarity is computed as

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$0

The attention center is then selected by

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$1

with confidence

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$2

and image-space center coordinates

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$3

Spot-IT then constructs an adaptive Gaussian attention mask whose spread depends on patch-selection confidence: $$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$4

$$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$5

The highlighted image is produced by blending the original page with a highlight color: $$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$6 where $$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$7 in the reported experiments (Thakkar et al., 7 Aug 2025). The modified image and the query are then passed to the MLLM with an instruction to answer preferentially from the highlighted region while retaining access to the full page if needed.

Methodologically, Spot-IT is notable for avoiding any architecture change to the downstream MLLM. Patch selection is external, Gaussian highlighting is external, and the system remains compatible with both open-source and closed-source multimodal models. The data summary explicitly characterizes the approach as simple, effective, zero-shot, and training-free (Thakkar et al., 7 Aug 2025).

5. Empirical findings, baselines, and failure modes

NiM evaluates OCR-based pipelines, direct image-based DocVQA, chain-of-thought prompting, and multiple MLLMs including Llama-3.2-VL, Qwen2-VL, GPT-4o, and Gemini-1.5-Flash (Thakkar et al., 7 Aug 2025). The reported result is that Spot-IT improves all tested models and datasets while leaving overall benchmark difficulty substantial.

On NiM, GPT-4o improves from EM 0.38 to 0.46 and from F1 0.48 to 0.56 with Spot-IT. On ArxiVQA, GPT-4o improves from 0.52 to 0.60 accuracy. Human performance on the NiM set is reported as 63% EM and 70% F1, but with much longer response time than models (Thakkar et al., 7 Aug 2025). The benchmark therefore exposes a clear gap between current MLLMs and human fine-grained document search, while also showing that targeted visual focus can close part of that gap.

The error analysis partitions failures into patch selection (26%), formation (25%), retrieval (22%), dataset (19%), and LLM faults (8%) (Thakkar et al., 7 Aug 2025). This decomposition is significant because it separates localization failure from answer-generation failure. The summary also states that MLLMs often fail even when given the correct region, indicating that the problem is not exhausted by better patch retrieval alone.

A similar pattern appears in VisualNeedle. There, no-tool accuracy stays below $$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$8, the best tool-enabled model reaches 56.01%, and human majority-vote accuracy is 63.00%. Under crop-black ablation, the best tool-enabled model drops to 12.00%, which shows that success depends on genuine intermediate visual evidence rather than on the ritual of tool use alone (Chen et al., 25 May 2026). This suggests that NiM’s reported gains from visual focus mechanisms should be interpreted as evidence-sensitive improvements rather than as generic prompting effects.

6. Relation to adjacent “needle” literatures

The phrase “finding needles” is used across several technical literatures, but NiM has a specific meaning. In document QA, the “needle” is a minute, decisive local clue. In medical image analysis, by contrast, the needle may be a literal surgical instrument. For retinal surgery images, a YOLOv5-based method predicts a bounding box, one of four tip-position classes (LT, LB, RT, RB), and a confidence score; it reports an average Euclidean distance of 4.80 between detected and actual tip position and an average angle error of 0.85 degrees across all test sets (Xu et al., 2023). In pelvic seed implant brachytherapy, multi-needle localization is reformulated as tip-handle detection and matching using an anchor-free HRNet-based detector and a greedy matching and merging method; the reported 3D tip MAE is $$\text{Fine-Granularity} = \frac{\text{Area of Relevant Region}}{\text{Total Image Page Area}} < 0.05$$9 mm, with precision $2\times2$0% and F1-score $2\times2$1% for 3D needle localization (Xiao et al., 22 Sep 2025). These works share the language of localization, but they target geometric instrument recovery rather than question-conditioned document reasoning.

A second adjacent literature concerns tiny objects in large images. One study formalizes difficulty through the Object to Image (O2I) ratio and reports that required training data scale rapidly with the inverse of O2I; it further observes that matching the receptive field to expected object size is beneficial and that the choice of global pooling operation matters at very low signal-to-noise ratios (Pawlowski et al., 2019). Another study uses variable-resolution sampling and iterative re-centering inspired by human vision, finding that variable-resolution models, particularly multi-channel models, can outperform a full-resolution baseline using only 5\% of the samples (Gorodissky et al., 2018). NiM differs in modality and supervision, but these results are relevant because they isolate the computational consequences of sparse, spatially localized evidence.

A third neighboring direction emphasizes discovery rather than prediction. In transductive active labeling for ecology, the central challenge is described as finding examples of rare classes embedded within dense regions of abundant classes, and sampling difficulty is quantified by

$2\times2$2

with a hybrid stopping rule that combines predictive and discovery criteria (Kurinchi-Vendhan et al., 2 Jun 2026). A plausible implication is that NiM-like evaluation can also be interpreted as discovery-limited: the model must surface the decisive region before reasoning quality can matter.

Taken together, these neighboring literatures clarify what NiM contributes. It does not merely ask whether a model can classify a tiny object, retrieve an image containing a small instance, or detect a literal medical needle. It asks whether a multimodal model can recover and use the correct local evidence inside a complex document layout when the relevant region is less than $2\times2$3 of the page and the answer depends on faithful fine-grained inspection (Thakkar et al., 7 Aug 2025).