MarkIt: Training-Free Visual Markers for Precise Video Temporal Grounding

Abstract: Video temporal grounding (VTG) aims to localize the start and end timestamps of the event described by a given query within an untrimmed video. Despite the strong open-world video understanding and recognition ability of video language large models (Vid-LLMs), outputting precise temporal grounding information remains challenging, since explicit temporal cues are scarce in untrimmed videos, and query-relevant entities are hard to track consistently across the video timeline. In this paper, we present \MarkIt{}, a training-free framework that transforms an input video into a query-conditioned marked video, which empowers Vid-LLMs to generate more reliable temporal localization predictions. The core component of \MarkIt{} is an annotation-free query-to-mask grounding bridge (Q2M-Bridge). Given a natural-language query, it automatically derives a compact set of canonical subject tags through linguistic parsing and normalization, then maps these tags to query-conditioned instance masks using text-conditioned open-vocabulary segmentation. The bridge also embeds lightweight semantic instance markers and a persistent frame index into each frame, effectively transforming long-range temporal reasoning into explicit visual cues for Vid-LLMs. \MarkIt{} adopts an inference-time plug-and-play design, needs no modifications to Vid-LLM weights, and is fully compatible with supervised fine-tuning. Experiments conducted on multiple mainstream moment retrieval and highlight detection benchmarks demonstrate that \MarkIt {} achieves state-of-the-art results, delivering consistent temporal grounding improvements across a wide range of existing models.

• The paper introduces a training-free framework that injects explicit visual markers into videos, enabling precise temporal localization.
• It employs a Q2M-Bridge to extract key subject tags and overlay region masks with persistent frame indices, transforming implicit temporal inference into explicit visual cues.
• Empirical evaluations on benchmarks like Charades-STA and ActivityNet show significant mIoU improvements, reducing false positives and narrowing event spans.

MarkIt: Training-Free Visual Markers for Precise Video Temporal Grounding

Overview

"MarkIt: Training-Free Visual Markers for Precise Video Temporal Grounding" (2604.25886) addresses the long-standing challenge of video temporal grounding (VTG), specifically, the precise localization of start and end timestamps of events queried in natural language within untrimmed videos. The core issue in VTG is the scarcity of highly explicit temporal cues in untrimmed video content and the need for consistent tracking of query-relevant entities. Although Video-Language Large Models (Vid-LLMs) have demonstrated strong performance in video understanding, they persistently underperform on precise temporal localization due to these limitations.

MarkIt introduces a training-free, plug-and-play framework that enhances input video streams with explicit, query-conditioned visual markers. The innovation lies in injecting lightweight region masks and persistent frame index markers into the video, making temporal reference and query-entity correspondence explicit. This design allows any downstream Vid-LLM to leverage these cues without model re-training or weight adaptation, thereby transforming temporal localization from an implicit reasoning problem to explicit visual reading.

Methodology

MarkIt is architected around a two-stage process:

1. Query-to-Mask Grounding Bridge (Q2M-Bridge): Given a natural-language query, Q2M-Bridge applies linguistic parsing and normalization to extract canonical subject tags (typically nouns or persons described in the query). A text-conditioned open-vocabulary segmentation model then grounds these tags as region masks on every frame, forming compact semantic overlays.
2. Frame Index and Marker Rendering: Every frame is augmented with a persistent frame index (for explicit temporal reference) and the query-conditioned region masks, annotated with text describing the extracted subject tags. Mask rendering uses a palette-driven approach with optimized opacities and contours to maximize saliency without introducing clutter. All overlays are generated at inference time, so the original Vid-LLM parameters remain untouched.

This process produces a "marked video" that can be processed by any off-the-shelf Vid-LLM for temporal grounding tasks.

Figure 1: System-level architecture: MarkIt extracts subjects and relations from query text, obtains region masks and frame indices for each frame, and injects these explicit visual cues into the video stream for cross-modal reasoning in a Vid-LLM.

Experimental Analysis

Evaluation spans several mainstream VTG benchmarks: Charades-STA and ActivityNet (moment retrieval) and QVHighlights (highlight detection), with protocols benchmarked under both training-free and supervised fine-tuning (SFT) settings on representative Vid-LLMs (Qwen2-VL-7B, LLaVA-OV-7B, InternVL2-8B, LongVA-7B-DPO).

Strong empirical improvements are observed:

• On Charades-STA, MarkIt boosts Qwen2-VL-7B's $mIoU$ from 7.9 to 41.1 (training-free) and LongVA's $mIoU$ from 14.6 to 43.9 (fine-tuned).
• Consistent, cross-model improvements manifest at all overlap thresholds—e.g., $[email protected]$ consistently rises by 9–40 percentage points depending on the backbone.
• On ActivityNet, MarkIt scales $mIoU$ from 12.5 to 33.3 (Qwen2-VL-7B), and up to 40.1 (LongVA-7B-DPO, fine-tuned)—closing the gap to VTG-specialized models via input markerization alone.
• On QVHighlights, MarkIt enables a 4–7 $mAP$ boost (training-free/general Vid-LLMs), outperforming prior prompting schemes.

Ablation studies confirm the centrality of subject-centric tag extraction, mask rendering style, region saliency, and text index overlay design. Notably, moderate palette opacities and boundary contours outperform high-opacity or excessively vivid colors, maximizing the informativeness of the overlays without interfering with model focus.

Qualitative Observations

MarkIt reliably narrows prediction spans to ground truth boundaries and reduces false positives, especially in scenarios with occlusions or multiple co-occurring entities. It demonstrates superior exclusion of background distractors compared to task-specific approaches (e.g., TimeChat, VTimeLLM).

Figure 2: On ActivityNet, MarkIt produces temporal spans that better match ground truth event boundaries than leading Vid-LLM baselines.

Theoretical Implications and Future Directions

MarkIt's design decouples semantic grounding and temporal localization by "externalizing" reference signals into the input, transforming the task from implicit temporal inference to explicit, region- and time-indexed identification. This exposes the limitations of current Vid-LLM architectures—most notably, their inability to internalize robust spatio-temporal correspondence—and provides a systematic channel for controllable, query-conditioned visual prompting.

This paradigm prompts several research directions:

• Integrated Marker and Prompt Co-design:

Adaptive marker generation in tandem with dynamic prompt engineering, facilitating open-vocabulary generalization and compositional event queries.

• Beyond Frame-Level Markers:

Incorporating temporal or trajectory-based markers (tracking entities across multiple frames, modeling persistence or transitions).

• Model-Driven Marker Learning:

Exploring joint optimization protocols where Vid-LLMs learn to anticipate or leverage marker semantics during training, potentially closing the gap between explicit prompting and end-to-end representation learning.

• Application to Other Spatio-Temporal Tasks:

Extension to dense captioning, video question answering, or event segmentation, where grounding of linguistic constructs to video is similarly ambiguous.

Conclusion

MarkIt demonstrates that comprehensive, training-free markerization of video content—injecting both explicit temporal and semantic correspondence cues—enables Vid-LLMs to achieve state-of-the-art VTG performance across both general and specialized settings. The framework is remarkably robust across architectures and does not require architecture or weight modification, positioning input markerization as a critical lever for actionable cross-modal grounding in large vision-LLMs.