NeMo: Montage-Based Sparse Target Detection

• Needle in a Montage (NeMo) is a design principle that creates structured montages embedding sparse target signals within semantically coherent distractors.
• It is applied in video-language tasks, referring image segmentation, and few-shot perception to rigorously test models on precise temporal and spatial grounding.
• Empirical results demonstrate that targeted negative sampling and geometry-aware aggregation substantially improve model performance and narrow the human-model gap.

Searching arXiv for papers using “Needle in a Montage / NeMo” across domains. In recent machine-learning literature, “Needle in a Montage” (NeMo) denotes a family of formulations built around the same structural problem: a sparse target must be identified, localized, or distilled from a montage of semantically related context. The term is formalized most explicitly as a video-language task for evaluating long-context recall and temporal grounding in VideoLLMs, but closely related uses appear in referring image segmentation, where the target object becomes a “needle” in a negative-mined mosaic, and in few-shot perception, where multi-view templates are compressed into a geometry-aware memory that isolates object-consistent signal from a montage of views (Hu et al., 29 Sep 2025, Ha et al., 2024, Jung et al., 4 Feb 2026). A complementary line of work in ultrasound needle segmentation treats montage-style temporal aggregation as an observation mechanism that can be stabilized by learned motion filtering, further broadening the concept’s methodological scope (Goel et al., 2023).

1. Core idea and terminological scope

Across the cited works, a montage is not an arbitrary collage. It is a deliberately structured aggregation of related clips, images, or views designed to preserve semantic plausibility while embedding substantial distractor content. The “needle” is correspondingly not a synthetic outlier but a target event, instance, or object-consistent representation that remains embedded in the same visual fabric as its context.

Context	Montage construction	Needle/target
Video-language understanding	Concatenation of loosely related short clips from the same source	Scene-level event or specific object instance requiring temporal grounding
Referring image segmentation	2×2 mosaic with 1 positive tile and 3 negative tiles	Ground-truth referent described by the expression
Few-shot perception	Multiple RGB template views aggregated into a reusable memory	Object-consistent geometry and semantics distilled into a sparse point cloud
Ultrasound needle analysis	Short temporal window or montage-like observation aggregate	Needle observation used to improve localization and segmentation

This recurring structure suggests that NeMo is best understood as a design pattern rather than a single task definition. In all three primary instantiations, the method or benchmark becomes difficult precisely because distractors are semantically adjacent: negative scenes come from the same TV program, negative RIS tiles are selected by multimodal similarity, and multi-view object templates differ mainly by viewpoint rather than category (Hu et al., 29 Sep 2025, Ha et al., 2024, Jung et al., 4 Feb 2026).

A common misconception is to equate NeMo with conventional “needle in a haystack” insertion. The video-language formulation explicitly rejects variants that insert irrelevant static elements such as text overlays or out-of-domain images, arguing that such settings under-challenge modern VideoLLMs because the inserted content does not live within the video’s semantic fabric or consistent shooting style (Hu et al., 29 Sep 2025). The RIS variant makes the same point operationally by calibrating negative difficulty so that distractors are neither too ambiguous nor too trivial (Ha et al., 2024).

2. Formalization in video-language understanding

The most explicit definition of Needle in a Montage appears in “NeMo: Needle in a Montage for Video-Language Understanding” (Hu et al., 29 Sep 2025). There, a montage video is defined as a temporal concatenation of short clips from the same source, and the task is to identify the temporal bounds of target needle segments from a natural-language question. The formulation distinguishes single-needle questions, where $M=1$, from multi-needle questions, where $M>1$.

The paper defines a montage video as

$V = \{s_1, s_2, \ldots, s_K\},$

with segment durations and spans, and target needle segments

$N = \{s_1^*, \ldots, s_M^*\}$

with ground-truth intervals

$$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$

Given a natural-language query $q$, the model outputs predicted temporal intervals $\hat{I}$, optionally together with textual answers, but evaluation is performed via temporal spans rather than free-form answer matching (Hu et al., 29 Sep 2025).

Temporal grounding is evaluated using a discrete temporal IoU. For a time range $$T_{\text{range}} = [T_{\text{start}}, T_{\text{end}}]$$, the timestamp set is

$$S_{\text{range}} = \{t \in \mathbb{Z} \mid \lfloor T_{\text{start}} \rfloor \le t \le \lceil T_{\text{end}} \rceil \},$$

and for multi-interval predictions or annotations the paper forms unions of such sets. The resulting score is

$$tIoU = \frac{|S_{gt} \cap S_{pred}|}{|S_{gt} \cup S_{pred}|}.$$

The main evaluation measures are Recall@1x, $M>1$0, Recall@1x, $M>1$1, and Average mAP over thresholds $M>1$2 (Hu et al., 29 Sep 2025).

Two capabilities are central. The first is long-context recall: the model must retrieve semantic details embedded within tens of minutes to hours of video. The second is temporal grounding: the model must localize needles precisely in time, including under temporal qualifiers such as “before,” “then,” and “after.” The benchmark therefore targets weaknesses that are characteristic of current VideoLLMs: limited context windows, weak time-aware modeling, lossy frame sampling, and a bias toward image-level cues over temporal structure (Hu et al., 29 Sep 2025).

The benchmark also distinguishes scene needles from object needles. Scene needles are coarse-grained compositions of objects and interactions; object needles are fine-grained object instances. This distinction matters because scene grounding and object grounding place different demands on temporal reasoning and semantic specificity, especially in multi-needle montages (Hu et al., 29 Sep 2025).

3. Data generation, benchmark composition, and empirical findings

NeMoBench is constructed through a three-stage automated pipeline. Stage 1 performs authorized video curation from Phoenix TV and extracts a compact tabular representation. Raw videos are segmented into scenes with PySceneDetect. For each scene, the pipeline records time spans, subtitles, object tags, links to object tables, referring expressions from UniRef, tracking sequences from DEVA, and per-frame localization; object tags are produced via RAM++ (Hu et al., 29 Sep 2025).

Stage 2 generates NeMo data. A target scene $M>1$3 is selected subject to duration and visibility constraints: the scene lasts at least $M>1$4 seconds, and prominent objects are visible for at least $M>1$5 seconds with area approximately $M>1$6–$M>1$7 of the frame. GPT-4o with visual prompting generates scene descriptions, scene-needle questions, and concrete object descriptions. Negative scenes are then sampled from the same source so they do not contain the target object tag, and these are concatenated with $M>1$8 to form a montage that preserves style and semantics. GPT-4o is used again for self-verification to remove low-quality QA pairs (Hu et al., 29 Sep 2025).

Stage 3 progressively expands the dataset from short montages to medium and long montages, and from single-needle to multi-needle questions. Eligible single-needle scenes longer than $M>1$9 seconds can be split into several clips of at least $V = \{s_1, s_2, \ldots, s_K\},$0 seconds each, which are then redistributed across negatives to create multi-needle montages (Hu et al., 29 Sep 2025).

The resulting benchmark has two variants. NeMoBench-Full contains 31,378 automatically generated QA pairs from 13,486 videos. NeMoBench-Clean contains 2,053 manually verified QA pairs from 940 videos. In the clean split, short videos contribute 1,246 QA pairs with average duration 26.20 s and average needles per question 3.19; medium videos contribute 507 QA pairs with average duration 5.13 min and average needles per question 3.33; long videos contribute 300 QA pairs with average duration 29.68 min and average needles per question 3.25 (Hu et al., 29 Sep 2025).

The automation claim is supported by an annotation-efficiency study. Theoretical average time reduction is reported as $V = \{s_1, s_2, \ldots, s_K\},$1, and in a 144-QA study manual annotation took 15.9 hours versus 3.5 hours for NeMo-Auto cleaning, approximately $V = \{s_1, s_2, \ldots, s_K\},$2 faster. Relative to ManualAnno, NeMo-Auto reports question accuracy 82.50 versus 91.00, Recall@1x, $V = \{s_1, s_2, \ldots, s_K\},$3 of 86.65 versus 95.83, and Average mAP of 97.59 versus 99.33, supporting the paper’s characterization of near-human quality (Hu et al., 29 Sep 2025).

Empirically, the benchmark exposes a large human-model gap. On short montages, the best open-source model is Qwen2.5-VL-72B with Avg 29.69%, whereas Gemini-1.5-Pro-002 reaches Avg 70.34% and human experts reach Avg 93.98%. On medium montages, the best open-source result is 11.22% from Qwen2.5-VL-72B, compared with 56.85% for Gemini-1.5-Pro-002 and 90.53% for human experts. On long montages, open-source models are described as near-zero in practice, while Gemini-1.5-Pro-002 reaches Avg 59.80% and human experts 81.97% (Hu et al., 29 Sep 2025).

Several ablations are diagnostically important. Multi-needle montages are harder than single-needle montages. Prompting matters substantially: the “interleaved with timestamp” format outperforms “video first,” with Qwen-VL-Max short-object recall increasing from 28.76% to 54.46%, GPT-4o from 31.64% to 64.07%, and Gemini-1.5-Pro-002 from 35.36% to 65.83%. Larger LLMs help, but increased input frame capacity alone does not guarantee better grounding (Hu et al., 29 Sep 2025).

4. Negative-mined mosaic augmentation in referring image segmentation

A second major NeMo formulation appears in “Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation” (Ha et al., 2024). Here the task is RIS: given an image $V = \{s_1, s_2, \ldots, s_K\},$4 and referring expression $V = \{s_1, s_2, \ldots, s_K\},$5, predict a binary mask $V = \{s_1, s_2, \ldots, s_K\},$6 for the target object, with ground-truth mask $V = \{s_1, s_2, \ldots, s_K\},$7. NeMo is not a new architecture or loss; it is a data-only augmentation that transforms a standard training example into a difficult 2×2 mosaic consisting of one positive tile and three carefully mined negative tiles.

The negative mining procedure uses a pretrained multimodal alignment model. With CLIP-style encoders $V = \{s_1, s_2, \ldots, s_K\},$8 and $V = \{s_1, s_2, \ldots, s_K\},$9, text-image similarity is

$N = \{s_1^*, \ldots, s_M^*\}$0

The paper also distinguishes text-to-image relevance $N = \{s_1^*, \ldots, s_M^*\}$1 and image-to-image relevance $N = \{s_1^*, \ldots, s_M^*\}$2. Candidates that are too similar are excluded via an upper-bound threshold $N = \{s_1^*, \ldots, s_M^*\}$3, using the filter condition $N = \{s_1^*, \ldots, s_M^*\}$4, after which the remaining candidates are ranked and a top-$N = \{s_1^*, \ldots, s_M^*\}$5 pool is formed. Three negatives are sampled uniformly from this pool (Ha et al., 2024).

Difficulty calibration is central. If negatives are unrelated, the augmentation collapses toward standard single-image training. If negatives are near-duplicates, the mosaic can contain multiple valid referents or induce positional inconsistencies. The ablations therefore tune $N = \{s_1^*, \ldots, s_M^*\}$6, $N = \{s_1^*, \ldots, s_M^*\}$7, and the application probability $N = \{s_1^*, \ldots, s_M^*\}$8. The paper reports $N = \{s_1^*, \ldots, s_M^*\}$9, with the best values around $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$0–$$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$1 for RefCOCO and RefCOCO+, and $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$2 for G-Ref. It reports best $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$3 for G-Ref and RefCOCO+, and $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$4 for RefCOCO. The best $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$5 is around $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$6–$$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$7 (Ha et al., 2024).

The mosaic uses a fixed cross-point 2×2 layout. Each source image is resized to half the target mosaic width and height and placed into a quadrant. The positive image retains the referring expression $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$8; negative tiles are labeled as background. The paper studies a constrained placement strategy for positional words such as “left” and “right,” but reports that the unconstrained version with tuned $$I = \{[t_s^{*(1)}, t_e^{*(1)}], \ldots, [t_s^{*(M)}, t_e^{*(M)}]\}.$$9 and $q$0 performs similarly or better (Ha et al., 2024).

Quantitatively, NeMo improves oIoU across all reported models and datasets, with larger gains on more complex datasets. For LAVT, RefCOCO Val rises from 72.73 to 73.25, RefCOCO+ TestA from 68.38 to 69.95, G-Ref UMD Test from 62.09 to 64.95, and GRES Val from 57.64 to 65.35. For CRIS, RefCOCO Val rises from 66.68 to 68.66, RefCOCO TestB from 59.93 to 63.06, G-Ref UMD Val from 55.91 to 58.47, and GRES Val from 54.55 to 56.23. Average gains are reported as RefCOCO $q$1, RefCOCO+ $q$2, G-Ref $q$3, and GRES $q$4 (Ha et al., 2024).

Comparisons to other augmentations clarify the mechanism. On G-Ref with CRIS, baseline oIoU is 55.91/58.50 for Val/Test; YOLOv4 mosaic gives 56.22/58.55, CutMix 56.50/58.34, MixGen 53.62/55.85, and NeMo 58.47/59.07. The paper attributes the advantage to language-conditioned disambiguation rather than generic composition: YOLO-style mosaic, MixUp, and CutMix can remove or occlude referents or blur contextual relations, whereas NeMo creates hard negatives that specifically demand fine-grained cross-modal grounding (Ha et al., 2024).

5. Geometry-aware memory from a montage of template views

A third usage appears in “Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception” (Jung et al., 4 Feb 2026). In this paper, NeMO means “Neural Memory Object,” but the method is explicitly framed as distilling the informative object-specific signal from a montage of diverse template views into a compact memory. The method targets few-shot detection, segmentation, and 6DoF pose estimation for unseen objects using RGB images only (Jung et al., 4 Feb 2026).

The input consists of $q$5 unordered RGB template views of a target object, captured with unknown cameras and no provided pose or intrinsics, together with a query RGB image. The encoder extracts patch features with a DINOv2 ViT, aggregates them using a multi-view encoder with anchor-biased cross- and self-attention, samples an initial 3D point cloud $q$6 in normalized coordinates, and predicts an unsigned distance function $q$7 whose gradient projects points onto the object surface. For each point,

$q$8

The resulting sparse set $q$9, together with fused semantic features, constitutes the NeMO memory (Jung et al., 4 Feb 2026).

The decoder fuses the NeMO with the query image and predicts modal and amodal masks, a dense 2D-to-3D pointmap $\hat{I}$0, and a confidence map $\hat{I}$1. Pose is then estimated by RANSAC+PnP from confident 2D–3D correspondences, with optional ICP refinement. Importantly, the method does not require camera intrinsics or extrinsics to build the memory object from templates, and it does not require retraining or fine-tuning on target objects at inference (Jung et al., 4 Feb 2026).

The paper reports competitive and state-of-the-art benchmark results. For model-free detection AP, HOPEv2 reaches 0.411 and HANDAL 0.273, both described as state-of-the-art on the leaderboard. For model-free 6DoF AP using NeMO detections, HOPEv2 reaches 0.302 and HANDAL 0.235. For model-based detection AP, TUD-L reaches 0.623, YCB-V 0.602, and T-LESS 0.183; for model-based segmentation AP, TUD-L reaches 0.488, YCB-V 0.579, and T-LESS 0.169 (Jung et al., 4 Feb 2026).

The methodological relevance to the broader NeMo idea is direct. Attention across views suppresses view-specific clutter and background and amplifies object-consistent tokens, while the learned UDF sharpens geometry by guiding sampled points onto the surface. The montage is therefore not merely stored; it is transformed into an object-centric representation whose inference cost is essentially constant regardless of the number of template views after offline construction (Jung et al., 4 Feb 2026).

6. Related extensions, limitations, and conceptual boundaries

The ultrasound needle-segmentation paper “Motion Informed Needle Segmentation in Ultrasound Images” does not introduce a NeMo method, but it explicitly situates montage or sequence-based approaches as complementary to motion-informed filtering (Goel et al., 2023). In that discussion, NeMo-style montages combine multiple frames into a single representation to improve signal-to-noise and highlight temporally consistent needle features, whereas the proposed KF-inspired module maintains a parametric state of tip, orientation, and length and learns how to predict and update that state from image features (Goel et al., 2023).

This distinction is operationally important. Montage/NeMo is described there as observation-centric and typically without explicit state maintenance; the motion-informed KF-CNN is prior-centric, explicitly modeling temporal dynamics and uncertainty. The paper suggests integrating them by using NeMo outputs as measurements $\hat{I}$2 for the learned filter, allowing the filter to stabilize montage outputs, handle missed detections, and enforce geometric plausibility. The reported effect of the motion-informed module alone is a 15% reduction in tip localization error and an 8% reduction in length error relative to frame-only segmentation baselines (Goel et al., 2023).

Across domains, several limitations recur. In video-language understanding, automated data generation relies on GPT-4o for question creation and self-verification, so residual noise persists despite near-human quality; long-video API evaluation is also costly (Hu et al., 29 Sep 2025). In RIS, poor calibration of $\hat{I}$3 and $\hat{I}$4 can create false negatives or false positives, and mosaics may become unnatural for highly heterogeneous image domains (Ha et al., 2024). In few-shot perception, symmetry and texturelessness degrade performance, segmentation can exhibit boundary artifacts, and close amodal instances may merge (Jung et al., 4 Feb 2026). In ultrasound, severe occlusion, complete out-of-plane motion, and strong reverberation can deceive both montage-based observations and learned filters (Goel et al., 2023).

Taken together, these works establish NeMo less as a single benchmark name than as a recurrent research principle: preserve semantic realism in the distractor field, make the target sparse but valid within that field, and force the model to recover it through temporal grounding, language-conditioned disambiguation, or geometry-aware aggregation. This suggests a broader methodological lesson: performance gains arise not from adding arbitrary clutter, but from constructing montages whose distractors are close enough to require the intended reasoning mechanism while remaining distinct enough to preserve label validity (Hu et al., 29 Sep 2025, Ha et al., 2024, Jung et al., 4 Feb 2026).