OmniShotCut: Holistic Shot Detection

• OmniShotCut is a holistic shot boundary detection framework that jointly predicts shot boundaries and assigns intra- and inter-shot semantic relations.
• It leverages a dense video Transformer with shot queries and synthetic transition synthesis to improve precision over traditional CNN-based methods.
• Empirical results on OmniShotCutBench show marked improvements in IoU, jump accuracy, and range F1, demonstrating its effectiveness in handling subtle video transitions.

OmniShotCut is a holistic relational framework for Shot Boundary Detection (SBD), formulated as a structured prediction task that jointly determines shot ranges along with intra-shot and inter-shot semantic relations. Unlike traditional SBD methods, which primarily localize temporal shot boundaries based on low-level heuristics or 3D-CNN classifiers, OmniShotCut leverages a dense video Transformer architecture grounded in shot-level queries, enhances supervision via a fully synthetic transition synthesis pipeline, and measures performance against a modern, wide-domain benchmark with precise evaluation criteria (Wang et al., 27 Apr 2026).

1. Problem Formulation and Structured Relational Prediction

Conventional SBD seeks to partition a sequence of $F$ video frames $X = \{x_1, x_2, \dots, x_F\}$ into a sequence of non-overlapping, temporally consecutive shots

$$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$

where each interval $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$ constitutes a temporally coherent shot. Traditional SBD systems are limited by producing only interval annotations without semantic interpretation, frequently missing abrupt discontinuities ("sudden jumps"), and suffering from imprecise labeling—especially for gradual transitions.

OmniShotCut generalizes SBD to holistic relational prediction. Each shot $s_i$ is endowed with:

• An intra-shot relation $$r^{\mathrm{intra}}_i \in \mathcal{C}_{\mathrm{intra}}$$ (with $|\mathcal{C}_{\mathrm{intra}}| = 8$), annotating the transition type within the shot (e.g., vanilla, dissolve, wipe, etc.).
• An inter-shot relation $$r^{\mathrm{inter}}_i \in \mathcal{C}_{\mathrm{inter}}$$ ($|\mathcal{C}_{\mathrm{inter}}| = 3$ plus a special "new-start" label), specifying the relation between the shot and its predecessor (hard-cut, sudden-jump, or transition).

The full output is the set $$\{ (t_i^{\mathrm{start}}, t_i^{\mathrm{end}}, r^{\mathrm{intra}}_i, r^{\mathrm{inter}}_i) \}_{i=1}^N$$, enabling structured prediction of both shot boundaries and their semantic context.

2. Shot-Query Transformer Architecture

The backbone of OmniShotCut is a Transformer-based architecture that processes spatio-temporal video evidence and predicts structured shot intervals with relational labels.

2.1 Shot Query Mechanism

A learnable bank of $X = \{x_1, x_2, \dots, x_F\}$0 shot queries $X = \{x_1, x_2, \dots, x_F\}$1, $X = \{x_1, x_2, \dots, x_F\}$2, is used to represent and attend to evidence for each candidate shot. Decoding is terminated via a learned "end-of-shots" token.

2.2 Dense Video Transformer

• Input encoding: Each input frame $X = \{x_1, x_2, \dots, x_F\}$3 is embedded via a ResNet, yielding a feature tensor $X = \{x_1, x_2, \dots, x_F\}$4.
• 3D positional encoding: Sinusoidal embeddings of $X = \{x_1, x_2, \dots, x_F\}$5 are applied, retaining spatio-temporal arrangement.
• Transformer encoder: Composed of $X = \{x_1, x_2, \dots, x_F\}$6 layers of multi-head self-attention mapping frame tokens to contextualized representations.
• Transformer decoder: Applies self-attention across shot queries and cross-attention between shot queries and encoded video features, yielding per-shot contextual embeddings $X = \{x_1, x_2, \dots, x_F\}$7.

2.3 Prediction Heads and Training Objective

Each finalized shot query $X = \{x_1, x_2, \dots, x_F\}$8 is mapped to three outputs:

1. Range head: Linear + softmax over $X = \{x_1, x_2, \dots, x_F\}$9 classes to predict shot end-frame index $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$0 (start taken as previous end); sidesteps heuristic thresholding and matching.
2. Intra-relation head: $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$1-way softmax $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$2.
3. Inter-relation head: $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$3-way softmax $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$4.

The total loss is a weighted sum of cross-entropies:

$$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$5

with $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$6, $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$7 in experiments.

3. Fully Synthetic Transition Synthesis Pipeline

OmniShotCut circumvents the imprecision, lack of diversity, and domain-constrained nature of legacy SBD labels by building a large-scale, parameterized synthetic dataset.

3.1 Clean-clip Curation

• Approximately $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$8 million raw Internet video clips are curated, filtered by frame rate, duration, and resolution.
• Semantic continuity is enforced using DINO embeddings with cosine similarity $$S = \left\{\, s_i = (t_i^{\mathrm{start}},\, t_i^{\mathrm{end}}) \,\right\}_{i=1}^N, \quad 0 = t_1^{\mathrm{start}} < t_1^{\mathrm{end}} < \cdots < t_N^{\mathrm{end}} = F$$9, and "motion strength" is estimated using CoTracker3 to exclude static content.
• Hierarchical K-means SSL clustering (27k clusters) on first-frame DINOv3 features ensures semantically coherent clip sampling, with deduplication threshold $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$0.

3.2 Transition Composition

Synthetic videos are composed by stochastic sampling:

• Number of clips $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$1, truncated to $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$2.
• Clip durations $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$3 s (or $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$4 s for "no-cut").
• 75% of clips drawn intra-cluster (else cross-cluster).
• Transition durations $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$5 s ($[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$6 s for whip-pan), transitions $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$7 frames default to hard cuts.
• 25% of videos have 28 dense, ultra-short clips.
• Hard cuts, in 90% of cases, are augmented with artificial "sudden jumps" via middle-cropping (24–40 frames, moderate-motion sources).
• Minor augmentations: subtitles (5%), lighting variation (7.5%).

3.3 Transition Families

Nine major transition families, with 30+ effect subtypes, are parameterized across space, time, and intensity. Families include:

• Hard cut & Sudden jump
• Dissolve (transparent, cross-blur, ripple)
• Wipe (directional, circular, mosaic, etc.)
• Push
• Slide
• Zoom (various modalities)
• Fade (black/white, dip)
• Doorway open
• Minor transitions

All transitions are parameterized to define precise, ground-truth boundaries and effect parameters.

4. OmniShotCutBench: Benchmark and Evaluation

A dedicated, expert-annotated dataset and protocol enables robust multi-domain SBD evaluation.

4.1 Dataset Specifications

• 114 videos, totaling approximately 110 minutes; each clip is $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$8 s, 480p @ 30 fps.
• Domains include vlog, anime, movie, concert, documentary, gameplay, sports, unboxing, screen, urban, tutorial, and more.
• Annotations include shot boundaries, intra/inter relations, and per-shot confidence scores; annotators are trained via editing tutorials.

4.2 Evaluation Metrics

Performance is assessed via:

• Range Precision, Recall, and F1: 2-frame tolerance, as per Cosmos ShotBench.
• Transition IoU: Temporal intersection-over-union for gradual transitions; tolerance dynamically adjusted by confidence.
• Sudden-Jump Accuracy: Requires exact frame-level match (zero tolerance).
• Intra-/Inter-relation Accuracy: Label correctness over matched shots.

Diagnostic breakdowns by transition family enable detailed error analysis.

5. Empirical Performance and Insights

OmniShotCut demonstrates significant improvements over state-of-the-art SBD methods (PySceneDetect, TransNet V2, AutoShot) on OmniShotCutBench:

Method	Transition IoU	Jump Acc	Range F1
PySceneDetect	0.183	0.416	0.754
TransNet V2	0.192	0.261	0.814
AutoShot	0.252	0.455	0.814
OmniShotCut	0.632	0.761	0.883

Additional relational scores: intra-relation accuracy = 0.959, inter-relation accuracy = 0.836.

Ablations reveal that:

• Classification-based range prediction (vs. DETR-style $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$9 regression) substantially improves frame-level accuracy and jump detection.
• SSL clustering during data curation is crucial; fully random pairing degrades most metrics.
• Densification of hard-cut sequences in synthesis pipeline augments detection of rapid transitions.

Qualitative analysis indicates that OmniShotCut boundaries align tightly with true effect onsets/offsets (especially for fades and dissolves), and the model recovers subtle sudden-jump artifacts missed by prior CNN-based detectors. Furthermore, predicted intra- and inter-relation labels yield explicit, interpretable transition semantics, facilitating downstream tasks such as video generation and scene understanding.

6. Significance and Broader Impact

OmniShotCut advances SBD research by:

• Framing the problem as holistic, relational prediction at the shot level.
• Realizing a Transformer-based model architecture with per-shot queries and dense spatio-temporal attention.
• Utilizing a fully synthetic, parameterized data pipeline to provide rich supervision.
• Establishing a broad, fine-grained evaluation benchmark with domain, effect, and confidence coverage.

This suggests the approach not only delivers state-of-the-art performance but also addresses interpretability, semantic grounding, and domain transfer challenges in modern video understanding (Wang et al., 27 Apr 2026).