A VideoMAE-v2 Approach to Zero-Shot Traffic Accident Anticipation

Abstract: Traffic accident anticipation -- predicting the likelihood of an imminent collision at every frame of a dashcam video -- is safety-critical yet difficult to scale, because collecting in-domain annotated accident footage for every deployment scenario is prohibitively expensive. We study this task under a zero-shot setting where no target-domain training data is available: the model must learn exclusively from a publicly available binary-labelled driving-accident dataset and generalise to unseen dashcam footage. We propose a framework that bridges the gap between the frame-level temporal risk estimation task and coarsely labelled binary accident datasets by coupling a VideoMAE-v2 backbone with a per-frame prediction head under a sliding-window protocol. Our method achieves 2nd place in the 2026 CVPR@AUTOPILOT Zero-Shot Accident Anticipation competition. Code is available at https://github.com/TimeSouth/zero-shot-taa-solution.

• The paper introduces a VideoMAE-v2 sliding-window framework that converts coarse accident labels into frame-level risk predictions and uses temporal aggregation for 150-frame dashcam clips.
• The paper achieves validation AP of 0.83, AUC of 0.86, and a private leaderboard score of 2.46234, placing second in the 2026 CVPR@AUTOPILOT competition.
• The paper shows that balanced dataset curation and full-context temporal modeling are essential, while removing either reduces the leaderboard score by 0.108 and 0.027 points, respectively.

Overview

This paper presents a zero-shot traffic accident anticipation (TAA) framework developed by Team BUPT MIC Lab for the 2026 CVPR@AUTOPILOT Zero-Shot Accident Anticipation competition, where it achieved 2nd place (2606.09542). The task requires producing a per-frame risk score $p_t \in [0,1]$ for every frame of a 150-frame dashcam clip, indicating the probability of a collision or near-miss occurring after that frame. The zero-shot constraint is strict: no target-domain training data is available, and the model must be trained exclusively on a publicly available binary-labelled accident dataset (Nexar) and generalise to unseen dashcam footage.

The authors identify two central technical obstacles. First, a task–data granularity mismatch: TAA demands dense frame-level supervision, while public accident datasets provide only coarse binary labels plus an event onset time $\tau$. Second, a long-clip / short-window modelling problem: VideoMAE-v2 accepts only 16 frames per forward pass, far fewer than the 150-frame evaluation clips, so a principled window-to-clip aggregation mechanism is required.

Method

The pipeline comprises three stages: dataset curation, sliding-window training and inference, and training-free test-time domain adaptation.

Dataset construction. Nexar videos are sliced into 5-second clips matching the competition's 150-frame format at 30 fps. For positive videos, the final 2 seconds before the event time $\tau$ are discarded so the model never observes the collision itself; frame-level labels are then derived from clip-level annotations ($y_t = 1$ throughout positive clips). A naive split is heavily skewed toward negatives, which would collapse cross-entropy to the trivial "always-negative" minimiser. The authors therefore apply a tiered balancing scheme: all positive clips are retained, negative clips cut from positive videos (informative pre-/post-event context) are retained, and clips from purely-negative videos are sub-sampled breadth-first with at most three per video until an approximately 1:1 positive-to-negative ratio is reached. Per-video capping preserves diversity in scenes, weather, and camera setups, which the authors report as important for cross-domain generalisation.

Training and inference. A VideoMAE-v2 Base encoder (86.2 M parameters) processes 16-frame windows sampled at 10 fps with stride 4. A lightweight per-frame head (0.9 M parameters) fuses global mean-pooled features with temporally upsampled token features to output per-frame risk probabilities, trained end-to-end with the discrete Exp-Loss of Bao et al., which exponentially down-weights positive-frame predictions far from the event so confidence is concentrated near temporal imminence. At inference, a 16-frame window slides with stride 1 across each test clip; predictions are aggregated by overlap averaging over covering windows, then linearly interpolated from the 10 fps sampling grid back to all 150 frame indices required by the evaluation.

Test-time domain adaptation. The source-trained model exhibits a systematic posterior shift on the target domain: relative rankings of frame-level predictions are preserved, but absolute confidences are compressed. Three label-free components address this:

• Temporal confidence aggregation: exploiting the prior that late frames carry stronger causal evidence, the last-frame prediction serves as the clip-level confidence anchor $p_{\text{final}}$, empirically having the highest signal-to-noise ratio.
• Temporal risk prior reconstruction: under the monotonicity prior that risk increases as time-to-event decreases, a dense risk curve is reconstructed from the anchor via a monotone basis function blending exponential and logarithmic schedules, with shape parameters $k \in [7,15]$ and $\mu \in [0.2,0.8]$ estimated from second-order statistics of the raw curve, and head-room factor $\beta = 0.998$ ensuring strict monotonicity.
• Prediction distribution alignment: the empirical median $m$ of $\{p_{\text{final}}\}$ across target clips is used to apply an order-preserving mapping that shifts the predicted distribution toward the expected decision boundary, analogous to test-time batch normalisation. Because the only target-set quantity used is a moment of the model's own outputs, the zero-shot constraint is respected.

A notable dependency here: the reconstruction step presupposes that the raw per-frame curve contains usable shape information. As the late-window ablation shows, if the input curve is flat by construction, this order-preserving post-processing cannot recover a rising risk profile.

Experiments

The full model (87.1 M parameters) is fine-tuned with AdamW (learning rates $\tau$0 backbone, $\tau$1 head), effective batch size 24 via gradient accumulation, 20 epochs, AMP, and gradient clipping, taking roughly 17 hours on a single V100. Checkpoint selection uses validation AP, AUC, and mTTA. Evaluation follows the official protocol: frame-level AP and AUC measure detection quality, while TTA and STTA measure early, stable anticipation; the private leaderboard reports a weighted composite score on unseen test data.

Two ablations test alternatives that appeared promising but degraded performance, each differing from the final pipeline in exactly one component:

Method	AP	AUC	Private LB
Proposed	0.83	0.86	2.46234
Late-window classification	0.66	0.69	2.43530
Mixed-dataset w/o balancing	0.78	0.60	2.35424

Late-window classification restricts input to the last 2 seconds of each clip, uniformly samples 16 frames, and produces a single scalar replicated across all 150 positions. It reaches validation AP = 0.66 and AUC = 0.69, and trails the final submission by −0.027 on the private leaderboard. Because its output has no temporal resolution by construction, the order-preserving post-processor cannot rebuild a rising risk curve from a flat one. The authors conclude that discarding the first 3 seconds of context costs more than the simpler supervision saves.

Cross-dataset mixed training without class balancing augments Nexar with DoTA positives but bypasses class balancing, yielding far more positives than negatives. The model degenerates into an "always-positive" predictor: validation AP remains deceptively high at 0.78 because AP rewards majority-class prediction under imbalance, but AUC collapses to 0.60—near the 0.5 chance level—showing the model can no longer rank positives above negatives. The private-leaderboard score drops by −0.108. Notably, the Exp-Loss time-weighting term provides no counterweight here since it only re-scales the positive loss. This result motivates treating dataset-level class balancing as a first-class pipeline component rather than a tunable hyper-parameter.

Limitations and open questions

Several constraints qualify the reported results. The method's validation metrics (AP 0.83, AUC 0.86) are computed on a held-out Nexar split—that is, in-domain—while the headline ranking comes from the private leaderboard composite score; the paper does not report AP/AUC/TTA decompositions on the unseen target domain, leaving the individual contribution of the test-time adaptation module unquantified. The distribution alignment step assumes the median of the model's own predictions is a reliable domain statistic; if the target domain's true accident prevalence differs substantially from what this moment implies, the recalibration could misalign the decision boundary, and the paper offers no sensitivity analysis on this assumption. Similarly, the temporal risk reconstruction fixes hyper-parameter ranges ($\tau$2, $\tau$3, $\tau$4) tuned on the source validation set, and their transferability across domains is not independently validated. Finally, the framework relies on a single source dataset; whether gains compound with additional balanced sources remains untested given the negative mixed-training result was confounded with the removal of balancing.

Conclusion

The paper demonstrates that a VideoMAE-v2 sliding-window classifier, combined with carefully class-balanced clip construction and a training-free test-time adaptation module exploiting temporal priors and prediction-distribution statistics, achieves competitive zero-shot traffic accident anticipation, placing 2nd in the 2026 CVPR@AUTOPILOT competition. The ablations carry the main methodological lesson: both full-context per-frame modelling and dataset-level class balancing are essential, as their absence causes measurable performance collapse (−0.027 and −0.108 leaderboard points respectively).