CoMET: Conditional Multi-Event Temporal Grounding

• CoMET is a unified task that defines the problem of exhaustively grounding events in long videos using compositional natural-language queries with temporal and spatial conditions.
• It establishes a benchmark with 600 long-form videos and rigorous evaluation metrics, including F1, MAE, and Rejection-F1, targeting precise event retrieval and count calibration.
• The CoMET-Agent framework leverages structured search-and-aggregate with hierarchical temporal graphs to improve grounding accuracy and address cross-moment reasoning challenges.

CoMET is a unified task, benchmark, evaluation protocol, and training-free agentic method for Conditional Multi-Event Temporal Grounding in long-form video. It formalizes the setting in which a model must identify every event instance in a long video that satisfies a compositional natural-language query containing temporal and spatial conditions, return precise temporal intervals for all valid instances, report the exact count, and correctly output the empty set for negative queries with no matches (Zou et al., 13 Jun 2026). In contrast to single-moment temporal grounding, unconditioned counting, or open-ended video question answering, CoMET requires exhaustive retrieval under compositional constraints and evaluates counting, grounding, and negative-query recognition jointly.

1. Formal task specification

CoMET takes as input a long-form video $V$ of duration $T$ and a compositional natural-language query $Q$ specifying temporal conditions $C_t$ and spatial conditions $C_s$. The temporal conditions are drawn from a small set of relations—causal, sequential, synchronous, bounded—while the spatial conditions constrain visual attributes—static, dynamic, identity. The output is a set of grounded temporal segments,

$P(V,Q)=\{(s_i,e_i)\}_{i=1}^{N},$

where each interval $(s_i,e_i)$ is the start and end of a qualifying event instance, and the cardinality $N=|P(V,Q)|$ is the exact count. Negative queries must be answered by the empty set (Zou et al., 13 Jun 2026).

The paper defines the task over the space of candidate intervals $E(V)$ through a spatial predicate $S(e;Q)\in\{0,1\}$ and a temporal predicate $T$0. The set of valid events is

$T$1

and CoMET requires exhaustive retrieval,

$T$2

In practice, the benchmark reports performance under a surrogate objective based on temporal overlap: $T$3 with $T$4 typically set to $T$5 and $T$6 the ground-truth set.

This formulation makes exhaustive retrieval the central requirement. A plausible implication is that CoMET treats omission and hallucination as equally structural errors: missing a valid interval harms set completeness and count calibration, while predicting a non-qualifying interval violates the query predicates themselves.

2. Benchmark construction and annotation protocol

CoMET-Bench contains 600 long-form videos averaging 33.8 minutes, 2,789 queries, and covers five domains: Sports, TV/Movie, Life Record, Knowledge, Surveillance (Zou et al., 13 Jun 2026). Its query distribution is explicitly compositional. Temporal condition types appear as Causal (11.6%), Sequential (11.9%), Synchronous (11.7%), and Bounded (19.7%); spatial condition types appear as Static (12.3%), Dynamic (17.2%), and Identity (15.6%). Negative queries comprise 26.6% of all queries, with the correct answer being [].

Queries are event-centric and must be visually verifiable with clear temporal boundaries; spatial-only counting is forbidden. The benchmark includes bounded windows such as “Between 00:45:00 and 00:46:30…” and relational operators including “after,” “while,” “because,” and “within.” Representative examples include a sports query requiring the localization of every successful “and-one,” a streaming/editing query asking for each transition from b-roll back to the streamer within a specified interval, and a surveillance query in which the correct answer is the empty set because no identity-constrained event occurs.

The dataset is explicitly long-horizon and dense. It includes 510 queries with eight or more events, 111 videos longer than 50 minutes, and 32 videos longer than 90 minutes. This rules out evaluation strategies that assume one salient moment per query or that treat retrieval as sparse by default.

Annotation follows a three-stage pipeline. First, query generation is performed by an MLLM under attribute constraints and then subjected to human verification for relevance, naturalness, and bias; spatial-only counting is rejected. Second, MLLM-assisted initial grounding is followed by human correction, including boundary tightening, addition of missed instances, and removal of spurious intervals. Third, a second annotator re-annotates, and pairs with tIoU agreement $T$7 are adjudicated by a senior reviewer. The reported quality statistics are stringent: 77.6% of candidate queries were rewritten, 31.7% of MLLM intervals were removed as spurious, 26.0% were newly added to recover misses, and only 6.5% of final pairs required adjudication (Zou et al., 13 Jun 2026).

These statistics indicate that automatic proposal generation is insufficient even before model evaluation. This suggests that CoMET is designed not merely as a harder retrieval benchmark, but as a benchmark whose annotation protocol is itself calibrated against systematic MLLM failure modes.

3. Evaluation protocol and metric design

CoMET’s evaluation protocol jointly measures instance-level grounding, set-level counting, and negative-query recognition rather than collapsing performance into a single scalar (Zou et al., 13 Jun 2026). Temporal overlap is measured with interval IoU: $T$8 with $T$9 for non-overlapping intervals.

For positive queries, greedy one-to-one matching at threshold $Q$0 defines true positives $Q$1, yielding

$Q$2

$Q$3

The protocol also reports mIoU, defined as the average, over each ground-truth instance, of the best IoU with any prediction.

Counting is evaluated at set level through

$Q$4

and Pearson correlation between predicted and ground-truth counts.

The most distinctive component is the treatment of negative queries. CoMET defines RejRate over negative queries and PosCov over positive queries: $Q$5 These are combined into

$Q$6

A model that always outputs [] attains $Q$7 but $Q$8, so Rejection-F1 = 0. The benchmark also reports FPR = 1 - RejRate.

This metric design directly addresses a recurrent misconception in long-video retrieval evaluation: that abstention on negatives can be handled independently of grounding quality. CoMET instead encodes abstention as a balanced recognition problem in which negative rejection is only meaningful if the model remains willing to ground positives.

4. Empirical performance and observed failure modes

The benchmark evaluates three broad families of methods: general-purpose MLLMs, agent-based pipelines, and grounding-specialized models (Zou et al., 13 Jun 2026). The MLLM suite includes GPT-5, Gemini 2.5 Pro, Gemini 3 Flash, InternVL3.5, Qwen3-VL, LLaVA-Video, LLaVA-OV1.5, MiMO-VL, Eagle2.5, LongVILA-R1. Agent-based pipelines include VideoMind, Vgent, T*, VideoARM. Grounding-specialized models include TRACE, DisTime, TimeLens, LITA.

Representative topline results show the gap between current capability and task requirements. The GPT-5 baseline reaches [email protected] = 10.1% and Rejection-F1 = 60.6% with FPR 5.3%. The Gemini 3 Flash baseline reaches [email protected] = 14.6% and Rejection-F1 = 61.5%, but with FPR 51.7%. Grounding-specialized models perform especially poorly in the compositional regime, with [email protected] < 5.4% and FPR > 93%, and TimeLens reported at FPR 100%.

Several systematic gaps emerge. Existing systems struggle with cross-moment temporal conditions—especially sequential, synchronous, causal relations—and with very long videos. Some agent pipelines cannot follow the required exhaustive timestamp-list output format and produce unparseable outputs. Scaling model size often improves count calibration but does not resolve grounding. Open-source 8–38B models remain far from proprietary baselines on grounding quality (Zou et al., 13 Jun 2026).

These findings clarify the nature of the task difficulty. CoMET is not hard only because the videos are long or the counts are large; it is hard because the query semantics require cross-interval composition, and because exhaustive retrieval amplifies boundary, recall, and hallucination errors simultaneously.

5. CoMET-Agent and the structured search-and-aggregate formulation

Building on the benchmark analysis, the paper introduces CoMET-Agent, a training-free framework that reformulates conditional multi-event temporal grounding as structured search-and-aggregate over a hierarchical temporal representation (Zou et al., 13 Jun 2026). All components are frozen.

The architecture has four principal components. The Planner Agent parses the query for bounded windows, selects a hyperparameter profile, and trims the video if the query specifies bounds. The Hierarchical Video Temporal Graph consists of an Event Graph $Q$9 built from DINOv2 features and change-point detection on inter-frame similarity, and an Action Graph $C_t$0 built from RAFT optical flow and PELT change-point detection. Edges capture temporal adjacency and semantically similar non-adjacent segments through weights combining cosine similarity and temporal decay. A Filter Agent prunes irrelevant event nodes before fine-grained expansion.

The Verifier Agent operates through iterative traversal. For a candidate action node $C_t$1, it constructs a perception context $C_t$2 by unioning its $C_t$3-hop neighbors whose edge weights exceed $C_t$4. It then verifies whether $C_t$5 satisfies both the spatial and temporal predicates in the query. On success it commits a memory entry

$C_t$6

to a Global Memory Bank $C_t$7. Online deduplication is performed via temporal NMS with IoU threshold $C_t$8, replacing lower-confidence duplicates. The Aggregator Agent consolidates $C_t$9 into final grounded intervals and the exact count, and resolves cross-moment identity consistency by comparing summaries $C_s$0 across entries.

The algorithmic loop is correspondingly simple in form: build $C_s$1, filter, expand to $C_s$2, initialize $C_s$3 and candidate set $C_s$4, then iterate up to $C_s$5 by sampling candidate nodes, expanding context, verifying, committing via temporal NMS, and removing covered nodes from $C_s$6. Complexity is described as favorable relative to exhaustive frame scanning: segmentation is $C_s$7 in sampled frames, graph node count is orders of magnitude smaller than frame count, and temporal-NMS deduplication is $C_s$8 per commit. With matched backbones, the framework is reported as ~2.8× faster than Vgent.

Empirically, this structural reformulation improves over single-pass prompting. With GPT-5 as backbone, CoMET-Agent reaches [email protected] = 16.2% and Rejection-F1 = 66.5%, improving [email protected] by +6.1 points over the GPT-5 baseline. With Gemini 3 Flash, it reaches [email protected] = 19.0% and Rejection-F1 = 68.1%, a +4.4 point gain. With Qwen3-VL-8B, it reaches [email protected] = 8.8% and Rejection-F1 = 57.4%, compared with a 3.4% single-pass baseline (Zou et al., 13 Jun 2026).

The paper attributes these gains to structural reasoning: the graph and iterative context expansion explicitly target cross-moment relations, while the memory bank persists identity and causal evidence across disjoint intervals. This suggests that CoMET-Agent’s gains are primarily representational and procedural rather than the result of additional learned capacity.

6. Relation to prior benchmarks, reproducibility, and open directions

CoMET is positioned against three adjacent but insufficient task families (Zou et al., 13 Jun 2026). Single-moment grounding benchmarks such as Charades-STA, QVHighlights, THUMOS14, MomentSeeker focus on one interval per query and rarely require compositional temporal conditioning. Counting benchmarks such as RepCount, Countix, OVR, EC-Bench quantify repetitions or enumerations but generally lack temporal conditioning and rarely require precise grounding of each counted instance. General video understanding QA benchmarks such as Video-MME, MLVU, LongVideoBench allow temporal conditions in question answering form but do not require exhaustive grounding. Negative queries exist in some recent single-moment retrieval work, but without conditioning. CoMET is described as uniquely unifying multi-event grounding, compositional temporal/spatial conditions, and negative queries under one evaluation framework with Rejection-F1.

The benchmark is designed for evaluation rather than supervised training. Videos are sourced from Video-MME, MLVU, CG-Bench, and a small YouTube subset; all use is non-commercial, research-only. Query–annotation pairs are released under CC-BY-NC-SA-4.0, and the dataset is gated to preserve upstream access controls. The paper does not define train/val/test splits for supervised training. A shared evaluation prompt returns strict JSON arrays of [start_seconds, end_seconds] for exhaustive intervals or [] for negatives, and the reported experiments use code and frozen backbones for reproducibility.

Failure analysis surfaces three open directions. Fine-grained entity tracking is needed because textual summaries are insufficient when many similar entities briefly co-occur; the paper proposes integrating multi-object tracking such as SAM 2 or SAMURAI so that nodes carry tracklets and queries can be verified against (interval, track-id). Position-uniform retrieval is motivated by recall degradation in mid-to-late video positions under similarity-based filtering; the proposed remedy is coverage-aware traversal with explicit temporal coverage regularization or per-bin budgets. Causal event pairing remains weak because models may ground plausible causes without verifying consequences; a cause–effect pairing module operating over action-graph node pairs is proposed.

Taken together, these directions indicate that CoMET is not merely a harder benchmark for long-video models, but a re-specification of the temporal grounding problem around exhaustive compositional retrieval. A plausible implication is that progress on CoMET will depend less on scaling generic video-LLMs in their current form than on adding explicit machinery for identity persistence, coverage control, and relational verification across distant moments.