Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video

Abstract: Grounding traffic accidents in real CCTV footage is a rare-event problem where training on labeled accident video is often prohibited, yet accurate joint localization in time, space, and collision type is required. We present a no-fine-tuning pipeline that elicits this joint output from frozen vision-LLMs through two ideas. First, a coarse-to-fine two-pass decomposition: a full-video pass at 1 fps produces a coarse (t, x, y, c) tuple, then a second pass at 5 fps within a +/- 3 s window refines time and location, with two deterministic confidence gates that revert to the coarse estimate on boundary hedges or edge-clamped coordinates. Second, a specialist role assignment: Qwen3-VL-Plus handles grounding, Gemini 3.1 Flash-Lite handles typing on a centered video clip. On the ACCIDENT@CVPR 2026 benchmark (2,027 real CCTV videos) we reach ACC^S = 0.539 (95% CI [0.525, 0.553]): +0.127 over the benchmark paper's best-of-baselines oracle (0.412), +0.143 over the strongest single-VLM baseline (Molmo-7B, 0.396), and +0.250 over the naive baseline (0.289). The VLM path uses up to three API calls per video (17% fall back to physics on API failures); the full run costs ~$20.

• The paper introduces a two-pass zero-shot grounding pipeline to accurately localize rare traffic events in terms of time, space, and collision type.
• It employs frozen vision-language models with a coarse-to-fine strategy and fallback techniques, ensuring robust performance without training.
• Experiments on CCTV videos demonstrate significant improvements in temporal, spatial, and type accuracy over traditional and single-shot baselines.

Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video

Problem Setting and Motivation

This paper addresses the challenge of rare-event grounding in traffic surveillance video, specifically the precise localization of traffic accidents in time, space, and collision type. The problem is framed by the ACCIDENT@CVPR~2026 benchmark (2604.09819), which prohibits training on real accident footage, enforcing a strict zero-shot evaluation regime. The target output of $(t^*, x^*, y^*, c^*)$ must be produced from unlabeled, real-world CCTV video, with evaluation based on the harmonic mean $\mathrm{ACC}^S$ across temporal, spatial, and categorical localization tasks. Existing approaches, including classical optical flow and bounding-box dynamics baselines, as well as single-shot VLM prompting (notably Molmo-7B), score well below human performance and struggle to balance all three subtasks without supervised fine-tuning.

Methodology: Coarse-to-Fine Two-Pass Grounding and VLM Role Assignment

The principal methodological innovation is a two-pass, coarse-to-fine grounding pipeline leveraging frozen vision-LLMs (VLMs) in a fully training-free protocol. The pipeline operates as follows:

1. Pass 1 (Coarse Grounding): The initial pass runs Qwen3-VL-Plus at 1 fps over the entire video, producing a coarse $(t_1, x_1, y_1, c_1)$ tuple. Frames are annotated with absolute timestamps, and the prompt translates discrete frames into candidate impact moments and spatial locations. On API failure (17%), a physics-based fallback (YOLO+ByteTrack) is employed.
2. Pass 2 (Fine Local Refinement): A 6s window ($\pm 3$\,s around $t_1$) is sampled at 5 fps and 1024 px for finer temporal and spatial resolution. Pass~2 prompts for exact impact time (to 0.1s) and location. Two deterministic gates are imposed:
   • Gate 1 (Temporal Fallback): If the fine estimate is missing or boundary-adjacent, revert to Pass~1 time.
   • Gate 2 (Spatial Merge): If fine coordinates are within margin, accept; otherwise, revert to coarse spatial location.
3. Specialist Type Classification: Collision typing is reassigned from Qwen3-VL to Gemini~3.1 Flash-Lite, using a spatially cropped 5s clip centered on $t^*$. On API failure, fallback to $c_1$.

(Figure 1)

Figure 1: The two-pass zero-shot grounding pipeline assigns temporal-spatial grounding to Qwen3-VL-Plus and type classification to Gemini~3.1, incorporating confidence gates for fallback.

This architecture exploits native timestamp alignment and grid coordinate output capabilities in modern VLMs while avoiding captioning-then-LLM or trained hierarchy protocols. API inference cost is $\sim$\$0.01 per video, enabling extensive benchmarking.

Experiments and Results

Evaluation on 2,027 real CCTV videos from ACCIDENT@CVPR~2026 reveals that the proposed pipeline achieves $\mathrm{ACC}^S{=}0.539$ (CI $[0.525, 0.553]$), substantially outperforming the benchmark paper's best-of-baselines oracle ($\mathrm{ACC}^S$0), strongest single-VLM baseline (Molmo-7B, $\mathrm{ACC}^S$1), and naive baseline ($\mathrm{ACC}^S$2). Temporal localization ($\mathrm{ACC}^S$3) is competitive, spatial localization ($\mathrm{ACC}^S$4) trails Molmo-7B but is robust, and type classification ($\mathrm{ACC}^S$5) is the highest reported.

Ablation analysis demonstrates that Gemini type classification improves $\mathrm{ACC}^S$6 by $\mathrm{ACC}^S$7 and Pass~2 refinement increases $\mathrm{ACC}^S$8 and $\mathrm{ACC}^S$9. The cumulative gain over Pass~1 is $(t_1, x_1, y_1, c_1)$0 in $(t_1, x_1, y_1, c_1)$1.

Failure Diagnostics and Analysis

The paper conducts a detailed failure mode analysis with the following key findings:

• Temporal Bias: Pass~1 predictions exhibit a right-skewed late bias (+1.55s mean), favoring post-collision frames rather than contact moments. Temporal MAE grows sharply with video length, indicating a need for motion-adaptive sampling.

Figure 2: Failure diagnostics across time bias, MAE vs. video length, type confusion, and ensemble oracle performance.

• Type Confusion: The confusion matrix reveals substantial collapse between head-on and t-bone (79%), and sideswipe and rear-end (39%), attributable to monocular depth ambiguity and web-corpus label bias. Head-on and sideswipe classes exhibit extremely low per-type accuracy ($(t_1, x_1, y_1, c_1)$2).
• Fallback Contribution: Physics-based fallback on Pass~1 API failures (17%) scores $(t_1, x_1, y_1, c_1)$3 $(t_1, x_1, y_1, c_1)$4, outperforming all classical baselines and indicating fallback robustness.
• Oracle Ensemble: An oracle combining YOLO+physics and VLM predictions reaches $(t_1, x_1, y_1, c_1)$5\,s temporal MAE, a $(t_1, x_1, y_1, c_1)$6 reduction.

Theoretical Implications and Future Directions

This study empirically validates the viability of training-free, zero-shot event grounding with frozen VLMs in challenging, privacy-constrained settings. The two-pass decomposition demonstrates that coarse temporal coverage and refined salience-focused analysis are synergistic. Specialist role assignment across VLMs for task separation is shown to mitigate VLM idiosyncrasies (e.g., poor typing in Qwen). The deterministic confidence gates, while simple, are robust and transferable across synthetic and real-world domains.

Practical deployment implications include immediate applicability for first-notice-of-loss workflows and post-incident analytics, subject to human-in-the-loop validation given temporal and type ambiguity. The full pipeline cost and latency are within operational tolerances for batch CCTV analysis.

Future work should explore motion-adaptive sampling to address late bias, integrate physics-VLM fusion for further temporal precision, and refine spatial grounding in challenging monocular views. Open-world deployment requires robust binary accident detection upstream of grounding. Continued advances in VLM timestamp modeling may eventually subsume the two-pass regime.

Conclusion

This paper presents a high-performing, zero-shot pipeline for temporal-spatial grounding and typing of rare traffic events in surveillance video, instantiated on ACCIDENT@CVPR~2026. The two-pass coarse-to-fine strategy and specialist VLM role assignment yield strong gains over prior approaches, with robust fallback diagnostics and cost-effective deployment. Continued progress will depend on adaptive temporal modeling and hybrid physics-LLM fusion.