NeMoBench: Video-Language Temporal Grounding

• NeMoBench is a video-language benchmark that evaluates models' ability to recall long video contexts and precisely pinpoint target sub-clips within extended montages.
• It features an automated three-stage data generation pipeline that creates diverse, distractor-heavy montages with open-ended QA pairs to rigorously test temporal grounding.
• The benchmark employs strict temporal IoU metrics and human baselines to assess both open-source and closed-source VideoLLMs on single- and multi-needle grounding tasks.

Searching arXiv for the specified paper and closely related benchmark context. NeMoBench is a video-language benchmark built around the Needle in a Montage (NeMo) task, introduced to evaluate Video LLMs (VideoLLMs) on two coupled capabilities: long-context recall and precise temporal grounding. Its central problem is an open-ended video question answering setting in which a model receives a long montage formed from many short clips and must identify the exact timestamp intervals where one or more target sub-clips appear. The benchmark is designed to move beyond standard recognition-style prompting and multiple-choice formats by requiring models to localize when relevant content occurs in extended, distractor-heavy video contexts (Hu et al., 29 Sep 2025).

1. Task formulation and benchmark scope

NeMoBench is centered on a “needle in a montage” evaluation. Rather than asking “What object do you see here?” or relying on multiple-choice answers, it fuses long-context recall and temporal grounding into a single task: given a long montage composed of dozens of loosely related short clips, a model must retrieve the exact time intervals where one or more video-relevant sub-clips appear. In its most general form, the NeMo task is a video-question-answering problem with temporal grounding (Hu et al., 29 Sep 2025).

Each test instance consists of a montage $V=\{c_1,c_2,\ldots,c_M\}$ formed by concatenating $M$ short video clips; one or more target sub-clips, or “needles,” each drawn from the montage’s source material but re-inserted at a new temporal location; a natural-language grounding question $Q$ that refers either to a specific object or to a broader scene; and an open-ended answer $A$ consisting of one or more timestamp intervals $\{[t_s,t_e]\}$. If a grounding question refers to $x$ distinct targets, the model must output $x$ intervals.

This construction distinguishes NeMoBench from benchmarks that test description quality without enforcing localization. The answer space is not a class label or a generated caption, but one or more temporally grounded intervals. A plausible implication is that the benchmark probes both retrieval over long visual contexts and alignment between language and temporal structure, rather than visual recognition alone.

2. Temporal grounding objective and evaluation metrics

The benchmark evaluates predictions with a set-based temporal Intersection over Union (tIoU) metric. Each interval $[T_{\text{start}},T_{\text{end}}]$ is represented as the integer set

$$S_{\text{range}}=\{\, t \in \mathbb{Z} \mid \lfloor T_{\text{start}} \rfloor \le t \le \lceil T_{\text{end}} \rceil \,\}.$$

Given a ground-truth set $S_{\text{gt}}$ and a predicted set $M$0,

$M$1

The evaluation computes $M$2, described as “Recall at 1 for $M$3 targets with threshold $M$4,” together with an average mean Average Precision (mAP) across thresholds from $M$5 to $M$6. For a question with $M$7 targets, let predictions and ground truths $M$8 be sets of integer timestamps as defined above. Then $M$9 is the fraction of ground-truth sets for which at least one top-1 prediction satisfies $Q$0 (Hu et al., 29 Sep 2025).

The primary reported metric is $Q$1, with $Q$2 also reported. The use of a relatively strict primary threshold makes the benchmark sensitive to temporal precision rather than coarse overlap alone. This suggests that approximate retrieval is insufficient: models are expected to localize relevant intervals with high fidelity.

3. Automated data generation pipeline

To scale the benchmark to tens of thousands of examples without labor-intensive manual annotation, the underlying work develops a three-stage automated data generation pipeline. The first stage, video curation and representation, collects high-quality, officially authorized TV program videos from Phoenix TV while ensuring that none overlap with public video-language training sets. Each raw video is segmented into scenes via a scene-boundary detector. For each scene clip $Q$3, a Scene Table stores start and end timestamps, object tags generated by an open-set image tagger, and subtitle text. For every tagged object, an Object Table records its tracked bounding-box sequence, a brief referring expression, and the object’s temporal span within the scene (Hu et al., 29 Sep 2025).

The second stage, NeMo data generation, begins by selecting a target scene and object. The pipeline chooses a clip of at least $Q$4 seconds containing at least one salient object that is visible for at least $Q$5 seconds and occupies $Q$6–$Q$7 of the frame. Grounding questions are then generated through visual prompting of a frozen multimodal model, GPT-4o, which produces a scene description, a scene-grounding question, and an object description. The corresponding temporal spans in the source video serve as raw answers, and object grounding questions are formed by template injection, for example, “When does {object description} show in the video?” The montage is created by sampling dozens of negative distractor scenes from the same video source while avoiding any mention of the target object or scene, then concatenating them with the target scene so that the original shooting style is preserved. Finally, the montage and generated QA pair are fed back into GPT-4o under a self-verification prompt, and any QA pairs flagged as invalid are discarded.

The third stage, progressive expansion, enlarges the task difficulty in two ways. Duration extension grows short montages of less than $Q$8 minutes into medium montages of $Q$9–$A$0 minutes and long montages of more than $A$1 minutes by inserting more distractors. Needle multiplication converts single-needle questions into multi-needle montages by segmenting the target scene into multiple sub-clips, each at least $A$2 seconds long, and inserting them as multiple targets. A final manual verification pass on the $A$3-example Clean subset corrects any remaining errors. The resulting pipeline synthesizes raw QA pairs with little human cost and is presented as reliable enough to support continuous updating with the latest videos.

4. Corpus composition and benchmark coverage

NeMoBench comprises two splits: NeMoBench-Full and NeMoBench-Clean. The Full split contains $A$4 QA pairs spliced from $A$5 montages, whereas the Clean split contains a rigorously human-verified core set of $A$6 QA pairs from $A$7 montages. Durations span from a few seconds to over one hour (Hu et al., 29 Sep 2025).

Split or category	QA pairs / montages	Duration
NeMoBench-Full	31,378 / 13,486	Few seconds to over one hour
NeMoBench-Clean	2,053 / 940	Human-verified core set
Clean short	1,246 / 657	Average $A$8 s
Clean medium	507 / 256	Average $A$9 min
Clean long	300 / 37	Average $\{[t_s,t_e]\}$0 min

Within NeMoBench-Clean, short montages of less than $\{[t_s,t_e]\}$1 minutes account for $\{[t_s,t_e]\}$2 QA pairs from $\{[t_s,t_e]\}$3 montages; medium montages of $\{[t_s,t_e]\}$4–$\{[t_s,t_e]\}$5 minutes account for $\{[t_s,t_e]\}$6 QA pairs from $\{[t_s,t_e]\}$7 montages; and long montages of more than $\{[t_s,t_e]\}$8 minutes account for $\{[t_s,t_e]\}$9 QA pairs from $x$0 montages. The QA pairs split roughly evenly between object versus scene grounding and between single-needle versus multi-needle settings. The montage duration histograms and category charts reported in the appendix are described as confirming broad coverage across video genres and object types.

The composition of the benchmark matters methodologically because it is not restricted to a narrow duration regime or a single grounding granularity. A plausible implication is that aggregate performance on NeMoBench conflates several distinct competencies: local object grounding, scene-level retrieval, robustness to distractors, and scaling behavior across increasing temporal context lengths.

5. Evaluation setup and empirical results

During evaluation, each model receives sampled frames at $x$1 FPS, up to $x$2 frames depending on model capacity, together with the grounding question. Two prompt styles are used. In the “video first” format, all frames are listed before the question. In the “interleaved with timestamp” format, each frame is interleaved with its timestamp before the question; closed-source models favor this latter style. The benchmark evaluates twenty models in total: two specialized grounding models, Moment-DETR and UniVTG; thirteen open-source VideoLLMs; and five closed-source models, namely Qwen-VL-Max, GPT-4o, Gemini-1.5-Flash-8B, Gemini-1.5-Flash-002, and Gemini-1.5-Pro-002. A $x$3 human baseline, with three experts per question, is used to establish an upper bound (Hu et al., 29 Sep 2025).

On short montages in NeMoBench-Clean, humans achieve $x$4. Among closed-source systems, Gemini-1.5-Pro-002 leads with $x$5, followed by GPT-4o at $x$6, Gemini-Flash-002 at $x$7, and Qwen-VL-Max at $x$8. Among open-source models, Qwen2.5-VL-72B performs best at only $x$9, and most open models remain below $x$0.

On medium montages, humans obtain $x$1. Gemini-Pro-002 reaches $x$2, Gemini-Flash-002 reaches $x$3, and GPT-4o drops to $x$4. The best open-source result is approximately $x$5 from Qwen2.5-VL-72B, while many models are near zero. On long montages, humans reach $x$6. Gemini-Pro-002 obtains $x$7, Gemini-Flash-002 obtains $x$8, and GPT-4o falls to $x$9. Open-source models are described as effectively near zero on long-form video.

Several ablation findings are reported. Montage duration strongly degrades performance. Multi-needle QA is consistently harder than single-needle QA. Adding explicit timestamp interleaving boosts top models by more than $[T_{\text{start}},T_{\text{end}}]$0 on short tests. Larger LLM backbones yield gains of $[T_{\text{start}},T_{\text{end}}]$1–$[T_{\text{start}},T_{\text{end}}]$2, suggesting that reasoning scale is key, whereas merely increasing frame capacity without stronger LLMs does not suffice. In addition, model rankings are nearly identical between the Clean and Full splits, indicating substantial consistency across the two benchmark versions.

6. Interpretation, limitations, and research directions

The reported findings portray NeMoBench as a stress test for VideoLLMs rather than a saturation benchmark. Open-source VideoLLMs struggle even on short montages and collapse to near zero on medium and long videos; the stated bottlenecks include limited context windows, naive grounding modules, and weak temporal reasoning. Closed-source VideoLLMs, including Gemini-1.5, GPT-4o, and Qwen-VL, make progress but still trail human experts by $[T_{\text{start}},T_{\text{end}}]$3–$[T_{\text{start}},T_{\text{end}}]$4 points in $[T_{\text{start}},T_{\text{end}}]$5. Temporal qualifiers in questions, such as “and then” and “before,” expose reasoning shortcuts, and even the top models falter on such cases. Multi-needle grounding further stresses models, with large gaps on multi-target questions (Hu et al., 29 Sep 2025).

These observations constrain how benchmark results should be interpreted. NeMoBench is not merely testing whether a model can recognize objects or summarize scenes; it is testing whether the model can retrieve and temporally localize semantically specified events within cluttered, extended visual streams. This suggests that improvements on generic video captioning or short-clip QA may not transfer directly to the benchmark’s long-context grounding regime.

The limitations and future directions presented with the benchmark are correspondingly specific. Proposed extensions include richer multimodal tasks using audio cues and subtitles, iterative or memory-augmented inference to better handle hour-long contexts, more fine-grained temporal reasoning probes such as ordering and causality, and learning objectives that explicitly align visual features with timestamped text. Taken together, these directions position NeMoBench as a benchmark for long-context video-language understanding whose principal contribution is to tightly weave retrieval over extended montages with timestamp-level localization.