---
title: 'CoMET: Conditional Multi-Event Temporal Grounding'
url: https://www.emergentmind.com/topics/comet-f8ae441d-2bef-4bcc-a0e7-f8c0df743aad
type: topic
---

# CoMET: Conditional Multi-Event Temporal Grounding

CoMET is a unified task, benchmark, evaluation protocol, and training-free agentic method for **Conditional Multi-Event Temporal Grounding in long-form video**. It formalizes the setting in which a model must identify **every** event instance in a long video that satisfies a compositional natural-language query containing temporal and spatial conditions, return precise temporal intervals for all valid instances, report the exact count, and correctly output the empty set for negative queries with no matches [2606.15320]. In contrast to single-moment temporal grounding, unconditioned counting, or open-ended video question answering, CoMET requires exhaustive retrieval under compositional constraints and evaluates counting, grounding, and negative-query recognition jointly.

## 1. Formal task specification

CoMET takes as input a long-form video $V$ of duration $T$ and a compositional natural-language query $Q$ specifying temporal conditions $C_t$ and spatial conditions $C_s$. The temporal conditions are drawn from a small set of relations—**causal, sequential, synchronous, bounded**—while the spatial conditions constrain visual attributes—**static, dynamic, identity**. The output is a set of grounded temporal segments,
\[
P(V,Q)=\{(s_i,e_i)\}_{i=1}^{N},
\]
where each interval $(s_i,e_i)$ is the start and end of a qualifying event instance, and the cardinality $N=|P(V,Q)|$ is the exact count. Negative queries must be answered by the empty set [2606.15320].

The paper defines the task over the space of candidate intervals $E(V)$ through a spatial predicate $S(e;Q)\in\{0,1\}$ and a temporal predicate $T(e,\mathcal{Z};Q)\in\{0,1\}$. The set of valid events is
\[
\mathcal{E}^\star(V,Q)=\Big\{\, e \in E(V)\;\Big|\;S(e;Q)=1\ \wedge\ T(e,\mathcal{Z};Q)=1\Big\},
\]
and CoMET requires exhaustive retrieval,
\[
\mathcal{P}(V,Q)=\mathcal{E}^\star(V,Q).
\]
In practice, the benchmark reports performance under a surrogate objective based on temporal overlap:
\[
\max_{\mathcal{P}\subseteq E(V)}\ \mathrm{F1@}\theta(\mathcal{P},\mathcal{G})\quad \text{s.t.}\quad \forall e\in\mathcal{P},\ S(e;Q)=1,\ T(e,\mathcal{Z};Q)=1,
\]
with $\theta$ typically set to $0.5$ and $\mathcal{G}$ the ground-truth set.

This formulation makes exhaustive retrieval the central requirement. A plausible implication is that CoMET treats omission and hallucination as equally structural errors: missing a valid interval harms set completeness and count calibration, while predicting a non-qualifying interval violates the query predicates themselves.

## 2. Benchmark construction and annotation protocol

CoMET-Bench contains **600 long-form videos averaging 33.8 minutes**, **2,789 queries**, and covers **five domains**: **Sports, TV/Movie, Life Record, Knowledge, Surveillance** [2606.15320]. Its query distribution is explicitly compositional. Temporal condition types appear as **Causal (11.6%)**, **Sequential (11.9%)**, **Synchronous (11.7%)**, and **Bounded (19.7%)**; spatial condition types appear as **Static (12.3%)**, **Dynamic (17.2%)**, and **Identity (15.6%)**. **Negative queries comprise 26.6% of all queries**, with the correct answer being `[]`.

Queries are event-centric and must be visually verifiable with clear temporal boundaries; spatial-only counting is forbidden. The benchmark includes bounded windows such as “Between 00:45:00 and 00:46:30…” and relational operators including “after,” “while,” “because,” and “within.” Representative examples include a sports query requiring the localization of every successful “and-one,” a streaming/editing query asking for each transition from b-roll back to the streamer within a specified interval, and a surveillance query in which the correct answer is the empty set because no identity-constrained event occurs.

The dataset is explicitly long-horizon and dense. It includes **510 queries with eight or more events**, **111 videos longer than 50 minutes**, and **32 videos longer than 90 minutes**. This rules out evaluation strategies that assume one salient moment per query or that treat retrieval as sparse by default.

Annotation follows a **three-stage pipeline**. First, query generation is performed by an MLLM under attribute constraints and then subjected to human verification for relevance, naturalness, and bias; spatial-only counting is rejected. Second, MLLM-assisted initial grounding is followed by human correction, including boundary tightening, addition of missed instances, and removal of spurious intervals. Third, a second annotator re-annotates, and pairs with **tIoU agreement $\le 0.7$** are adjudicated by a senior reviewer. The reported quality statistics are stringent: **77.6% of candidate queries were rewritten**, **31.7% of MLLM intervals were removed as spurious**, **26.0% were newly added to recover misses**, and **only 6.5% of final pairs required adjudication** [2606.15320].

These statistics indicate that automatic proposal generation is insufficient even before model evaluation. This suggests that CoMET is designed not merely as a harder retrieval benchmark, but as a benchmark whose annotation protocol is itself calibrated against systematic MLLM failure modes.

## 3. Evaluation protocol and metric design

CoMET’s evaluation protocol jointly measures instance-level grounding, set-level counting, and negative-query recognition rather than collapsing performance into a single scalar [2606.15320]. Temporal overlap is measured with interval IoU:
\[
\mathrm{IoU}\big([s_1,e_1],[s_2,e_2]\big)=
\frac{\min(e_1,e_2)-\max(s_1,s_2)}
{\max(e_1,e_2)-\min(s_1,s_2)},
\]
with $\mathrm{IoU}=0$ for non-overlapping intervals.

For positive queries, greedy one-to-one matching at threshold $\theta=0.5$ defines true positives $\mathrm{TP}_q$, yielding
\[
\mathrm{Precision@}\theta_q=\frac{\mathrm{TP}_q}{\hat N_q},\qquad
\mathrm{Recall@}\theta_q=\frac{\mathrm{TP}_q}{N_q},
\]
\[
\mathrm{F1@}\theta_q=
\frac{2\cdot \mathrm{Precision@}\theta_q\cdot \mathrm{Recall@}\theta_q}
{\mathrm{Precision@}\theta_q+\mathrm{Recall@}\theta_q}.
\]
The protocol also reports **mIoU**, defined as the average, over each ground-truth instance, of the best IoU with any prediction.

Counting is evaluated at set level through
\[
\mathrm{MAE}=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}|\hat N_q-N_q|,
\qquad
\mathrm{OBO}=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathbb{1}[|\hat N_q-N_q|\le 1],
\]
and **Pearson** correlation between predicted and ground-truth counts.

The most distinctive component is the treatment of negative queries. CoMET defines **RejRate** over negative queries and **PosCov** over positive queries:
\[
\mathrm{RejRate}=\frac{1}{|\mathcal{Q}^-|}\sum_{q\in\mathcal{Q}^-}\mathbb{1}[\hat N_q=0],
\qquad
\mathrm{PosCov}=\frac{1}{|\mathcal{Q}^+|}\sum_{q\in\mathcal{Q}^+}\mathbb{1}[\hat N_q>0].
\]
These are combined into
\[
\mathrm{Rejection\mbox{-}F1}=
\frac{2\cdot \mathrm{RejRate}\cdot \mathrm{PosCov}}
{\mathrm{RejRate}+\mathrm{PosCov}}.
\]
A model that always outputs `[]` attains $\mathrm{RejRate}=1$ but $\mathrm{PosCov}=0$, so **Rejection-F1 = 0**. The benchmark also reports **FPR = 1 - RejRate**.

This metric design directly addresses a recurrent misconception in long-video retrieval evaluation: that abstention on negatives can be handled independently of grounding quality. CoMET instead encodes abstention as a balanced recognition problem in which negative rejection is only meaningful if the model remains willing to ground positives.

## 4. Empirical performance and observed failure modes

The benchmark evaluates three broad families of methods: **general-purpose MLLMs**, **agent-based pipelines**, and **grounding-specialized models** [2606.15320]. The MLLM suite includes **GPT-5, Gemini 2.5 Pro, Gemini 3 Flash, InternVL3.5, Qwen3-VL, LLaVA-Video, LLaVA-OV1.5, MiMO-VL, Eagle2.5, LongVILA-R1**. Agent-based pipelines include **VideoMind, Vgent, T\*, VideoARM**. Grounding-specialized models include **TRACE, DisTime, TimeLens, LITA**.

Representative topline results show the gap between current capability and task requirements. The **GPT-5 baseline** reaches **F1@0.5 = 10.1%** and **Rejection-F1 = 60.6%** with **FPR 5.3%**. The **Gemini 3 Flash baseline** reaches **F1@0.5 = 14.6%** and **Rejection-F1 = 61.5%**, but with **FPR 51.7%**. Grounding-specialized models perform especially poorly in the compositional regime, with **F1@0.5 < 5.4%** and **FPR > 93%**, and **TimeLens** reported at **FPR 100%**.

Several systematic gaps emerge. Existing systems struggle with **cross-moment temporal conditions**—especially **sequential, synchronous, causal** relations—and with very long videos. Some agent pipelines cannot follow the required exhaustive timestamp-list output format and produce unparseable outputs. Scaling model size often improves count calibration but does not resolve grounding. Open-source **8–38B** models remain far from proprietary baselines on grounding quality [2606.15320].

These findings clarify the nature of the task difficulty. CoMET is not hard only because the videos are long or the counts are large; it is hard because the query semantics require cross-interval composition, and because exhaustive retrieval amplifies boundary, recall, and hallucination errors simultaneously.

## 5. CoMET-Agent and the structured search-and-aggregate formulation

Building on the benchmark analysis, the paper introduces **CoMET-Agent**, a **training-free** framework that reformulates conditional multi-event temporal grounding as **structured search-and-aggregate** over a hierarchical temporal representation [2606.15320]. All components are frozen.

The architecture has four principal components. The **Planner Agent** parses the query for bounded windows, selects a hyperparameter profile, and trims the video if the query specifies bounds. The **Hierarchical Video Temporal Graph** consists of an **Event Graph** $G_E=(V_E,E_E)$ built from **DINOv2** features and change-point detection on inter-frame similarity, and an **Action Graph** $G_A=(V_A,E_A)$ built from **RAFT optical flow** and **PELT** change-point detection. Edges capture temporal adjacency and semantically similar non-adjacent segments through weights combining cosine similarity and temporal decay. A **Filter Agent** prunes irrelevant event nodes before fine-grained expansion.

The **Verifier Agent** operates through iterative traversal. For a candidate action node $v_i$, it constructs a perception context $C(v_i)$ by unioning its $K$-hop neighbors whose edge weights exceed $\tau$. It then verifies whether $C(v_i)$ satisfies both the spatial and temporal predicates in the query. On success it commits a memory entry
\[
m=\langle \text{summary }S,\ \text{interval }(t_{\text{start}},t_{\text{end}}),\ \text{confidence }c\rangle
\]
to a **Global Memory Bank** $\mathcal{M}$. Online deduplication is performed via temporal NMS with IoU threshold $\gamma$, replacing lower-confidence duplicates. The **Aggregator Agent** consolidates $\mathcal{M}$ into final grounded intervals and the exact count, and resolves cross-moment identity consistency by comparing summaries $S$ across entries.

The algorithmic loop is correspondingly simple in form: build $G_E$, filter, expand to $G_A$, initialize $\mathcal{M}=\varnothing$ and candidate set $U=V_A$, then iterate up to $N_{\max}$ by sampling candidate nodes, expanding context, verifying, committing via temporal NMS, and removing covered nodes from $U$. Complexity is described as favorable relative to exhaustive frame scanning: segmentation is **$O(T)$** in sampled frames, graph node count is orders of magnitude smaller than frame count, and temporal-NMS deduplication is **$O(|\mathcal{M}|)$** per commit. With matched backbones, the framework is reported as **~2.8× faster than Vgent**.

Empirically, this structural reformulation improves over single-pass prompting. With **GPT-5** as backbone, **CoMET-Agent** reaches **F1@0.5 = 16.2%** and **Rejection-F1 = 66.5%**, improving F1@0.5 by **+6.1 points** over the GPT-5 baseline. With **Gemini 3 Flash**, it reaches **F1@0.5 = 19.0%** and **Rejection-F1 = 68.1%**, a **+4.4 point** gain. With **Qwen3-VL-8B**, it reaches **F1@0.5 = 8.8%** and **Rejection-F1 = 57.4%**, compared with a **3.4%** single-pass baseline [2606.15320].

The paper attributes these gains to structural reasoning: the graph and iterative context expansion explicitly target cross-moment relations, while the memory bank persists identity and causal evidence across disjoint intervals. This suggests that CoMET-Agent’s gains are primarily representational and procedural rather than the result of additional learned capacity.

## 6. Relation to prior benchmarks, reproducibility, and open directions

CoMET is positioned against three adjacent but insufficient task families [2606.15320]. **Single-moment grounding benchmarks** such as **Charades-STA, QVHighlights, THUMOS14, MomentSeeker** focus on one interval per query and rarely require compositional temporal conditioning. **Counting benchmarks** such as **RepCount, Countix, OVR, EC-Bench** quantify repetitions or enumerations but generally lack temporal conditioning and rarely require precise grounding of each counted instance. **General video understanding QA benchmarks** such as **Video-MME, MLVU, LongVideoBench** allow temporal conditions in question answering form but do not require exhaustive grounding. Negative queries exist in some recent single-moment retrieval work, but without conditioning. CoMET is described as uniquely unifying **multi-event grounding**, **compositional temporal/spatial conditions**, and **negative queries** under one evaluation framework with **Rejection-F1**.

The benchmark is designed for evaluation rather than supervised training. Videos are sourced from **Video-MME, MLVU, CG-Bench**, and a small **YouTube** subset; all use is **non-commercial, research-only**. Query–annotation pairs are released under **CC-BY-NC-SA-4.0**, and the dataset is gated to preserve upstream access controls. The paper does **not** define train/val/test splits for supervised training. A shared evaluation prompt returns strict JSON arrays of `[start_seconds, end_seconds]` for exhaustive intervals or `[]` for negatives, and the reported experiments use code and frozen backbones for reproducibility.

Failure analysis surfaces three open directions. **Fine-grained entity tracking** is needed because textual summaries are insufficient when many similar entities briefly co-occur; the paper proposes integrating multi-object tracking such as **SAM 2** or **SAMURAI** so that nodes carry tracklets and queries can be verified against `(interval, track-id)`. **Position-uniform retrieval** is motivated by recall degradation in mid-to-late video positions under similarity-based filtering; the proposed remedy is coverage-aware traversal with explicit temporal coverage regularization or per-bin budgets. **Causal event pairing** remains weak because models may ground plausible causes without verifying consequences; a cause–effect pairing module operating over action-graph node pairs is proposed.

Taken together, these directions indicate that CoMET is not merely a harder benchmark for long-video models, but a re-specification of the temporal grounding problem around exhaustive compositional retrieval. A plausible implication is that progress on CoMET will depend less on scaling generic video-language models in their current form than on adding explicit machinery for identity persistence, coverage control, and relational verification across distant moments.

Source: https://www.emergentmind.com/topics/comet-f8ae441d-2bef-4bcc-a0e7-f8c0df743aad