OmniShotCut: Holistic Shot Boundary Detection

• The paper recasts shot boundary detection as a structured relational prediction task, enabling simultaneous localization and semantic relation inference using shot queries.
• The proposed shot-query transformer leverages an encoder-decoder architecture with learnable queries to predict precise shot boundaries and diverse intra/inter-shot relations.
• A comprehensive synthetic transition pipeline and the OmniShotCutBench benchmark yield improved annotation precision and significant gains over previous SBD methods.

OmniShotCut is a holistic approach to Shot Boundary Detection (SBD), formulated as a structured relational prediction problem. It advances SBD by explicitly modeling not only shot boundaries but also intra-shot and inter-shot relations, leveraging a shot query-based Transformer architecture, a comprehensive synthetic transition generation pipeline, and a modern, professionally-annotated benchmark. This methodology enables state-of-the-art localization and interpretability, addressing longstanding challenges in detection accuracy, transition diversity, and label reliability (Wang et al., 27 Apr 2026).

1. Structured Formulation of Shot Boundary Detection

OmniShotCut recasts SBD from simple boundary localization to a structured relational task. Given a video of $F$ frames, the system predicts a sequence of $N$ non-overlapping shot intervals $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$ with $1 = s_1 \leq e_1 < s_2 \leq \dots < e_N = F$. Each shot is annotated with an intra-shot relation $r_i^{\mathrm{intra}}$ (type of content or transition inside the shot) chosen from $C_{\mathrm{intra}} = 8$ classes (e.g., vanilla, dissolve, wipe), and an inter-shot relation $r_i^{\mathrm{inter}}$ (transition between shots) drawn from $C_{\mathrm{inter}} = 3$ classes (hard-cut, gradual transition, sudden jump; marking the first shot as “new-start”).

The model $f_\theta$ takes input frames $X$ and outputs end-frame indices $N$0, intra scores $N$1, and inter scores $N$2. The objective function is a weighted sum of three cross-entropy losses: $N$3 with hyperparameters $N$4. Each term evaluates prediction accuracy for endpoints, intra labels, and inter labels respectively using ground-truth $N$5.

This structured formulation allows the joint estimation of shot segmentation and semantic/relational attributes, providing a richer and more interpretable model of video structure than traditional frame-level or segment-level SBD approaches.

2. Shot-Query Transformer Architecture

OmniShotCut adapts the encoder-decoder paradigm of DETR to dense video analysis using learnable “shot queries” that jointly predict both shot boundaries and relations. The core components are:

• Per-frame feature extraction: Each frame is processed by a ResNet-18 backbone, producing spatial features $N$6.
• Spatio-temporal encoding: All frame-level features are flattened and concatenated into $N$7, augmented by 3D (temporal and spatial) sinusoidal position embeddings. These are processed by $N$8 layers of multi-head self-attention with feed-forward networks.
• Transformer decoder with shot queries: $N$9 learnable queries $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$0 serve as prediction slots. Decoder layers sequentially alternate between self-attention and cross-attention with $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$1, followed by a feed-forward network.
• Prediction heads per query: Each decoder output provides:
  • Range head: predicts frame-index logits $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$2, yielding $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$3 via softmax over all frames.
  • Intra head: produces $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$4 via softmax over intra-shot classes.
  • Inter head: outputs $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$5 for inter-shot relations.

The architecture eliminates the need for Hungarian matching due to the direct classification of endpoints and introduces auxiliary losses at intermediate decoder layers for stability. This “shot query” mechanism (an instance of structured queries) enables simultaneous localization and relational inference in a single pass.

This approach is conceptually related to the Temporal Perceiver architecture, which leverages a limited, structured set of latent queries to compress and summarize long input sequences, and performs set prediction via transformer cross-attention without reliance on post-processing heuristics (Tan et al., 2022).

3. Fully Synthetic Transition Synthesis Pipeline

To circumvent the limitations of imprecise and low-diversity manual annotations, OmniShotCut introduces a fully synthetic transition generation pipeline, creating multi-shot videos with precise boundary control and annotation. The process comprises:

• Raw video curation: ~2.5M internet videos are filtered by quality; segments are extracted using DINOv3 embeddings for content homogeneity, and motion profiles are assessed with CoTracker3 for sudden-jump sources. Deduplication and clustering by SSL yield 27,000 clusters.
• Clip sampling: Each synthetic video samples a Poisson$\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$6 number of clips (clipped to $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$7), draws durations from specified normal distributions based on context, and places 75% of consecutive clips from the same cluster for semantic consistency.
• Transition synthesis: A mixture of transition types is applied to simulated boundaries:
  • Hard cut (35%), several forms of dissolve (totaling over 13%), diverse wipes, push, slide, zoom, fade, doorway, whip-pan, and sudden jumps. Transition durations and compositional parameters are drawn from designed distributions to maximize coverage.
  • Sudden jumps are implemented by randomly excising short frame spans with high probability on hard cuts.
• Offline augmentations: A proportion of videos receive overlay text and color/luminance perturbations to approximate real-world degradation and occlusion.

Each synthetic video is accompanied by complete ground-truth for indices and shot/transition labels. The design allows systematic creation of hundreds of transition variants with fine-grained annotation, improving label precision and transition diversity compared to human-labeled datasets.

4. OmniShotCutBench Benchmark and Evaluation Protocol

OmniShotCutBench is a modern benchmark dataset tailored for holistic and diagnostic SBD evaluation. It consists of 114 manually curated videos (110 minutes in total, 480p / 30fps) spanning a broad set of genres including vlogs, anime, films, concerts, documentaries, gameplay, and sports.

Annotation is performed by annotators trained on professional editing guidelines, with multi-round pilots and consensus protocols for ambiguous boundary cases, and a custom tool facilitating per-frame annotation and confidence scoring.

The evaluation protocol includes:

• Range Precision/Recall/F1: Predicted shot boundary is correct if within $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$82 frames of ground truth.
• Transition IoU: For gradual transitions, overlap between predicted and ground truth boundaries is evaluated using intersection-over-union, with tolerance modulated by annotation confidence.
• Sudden Jump Accuracy: Frame-level exact matching with zero tolerance.
• Intra/Inter relation accuracy: Relational label correctness is scored by matching each ground truth shot to the highest-IoU prediction.

This suite of metrics enables detailed analysis of SBD algorithms across both localization and semantic/relational axes.

5. Experimental Results and Analysis

Performance on OmniShotCutBench demonstrates substantial advances over previous SBD approaches. Table 1 provides key quantitative results:

Method	Trans. IoU	Sudden Jump Acc.	Range Prec.	Range Rec.	Range F₁	Intra Acc.	Inter Acc.
PySceneDetect	0.183	0.416	0.833	0.689	0.754	–	–
TransNet V2	0.192	0.261	0.913	0.734	0.814	–	–
AutoShot	0.252	0.455	0.849	0.782	0.814	–	–
OmniShotCut	0.632	0.761	0.898	0.858	0.883	0.959	0.836

Notable outcomes:

• Transition localization (IoU) by OmniShotCut more than doubles previous best results.
• Sudden jump detection accuracy reaches 76.1%, compared to less than 46% for baselines.
• Intra/inter-relational classification achieves over 83% accuracy.

Ablation studies reveal several important design sensitivities:

• $\mathcal{S} = \{ (s_i, e_i)\}_{i=1}^N$9+GIoU regression for range encoding slightly improves transition IoU but significantly degrades sudden-jump and one-frame precision.
• Sampling consecutive clips from semantically aligned clusters is vital; fully random sampling degrades all metrics.
• Increasing the prevalence of short, dense hard-cut transitions in the synthetic dataset improves sudden-jump recall and general range F1, better matching the real-world transition distribution.

6. Relation to Broader Work and Architecture Comparisons

OmniShotCut embodies a new paradigm in SBD by embracing structured prediction and holistic relational modeling. Prior transformer-based models such as Temporal Perceiver (Tan et al., 2022) achieve state-of-the-art generic boundary detection using latent query compression, explicit query partitioning (boundary/context), and alignment losses. The OmniShotCut shot-query transformer extends these principles:

• Structured queries serve as anchors for shots and their semantic relations.
• Joint range and relation predictions permit direct end-to-end mapping from frames to a segmented, semantically labeled video narrative.
• Synthetic data generation augments the training regime, systematically increasing annotation diversity and precision beyond levels attainable via manual curation.

A plausible implication is that further differentiation of query types (e.g., dedicated “cut-queries,” “dissolve-queries,” or higher-order relation queries as suggested in (Tan et al., 2022)) could further enrich structured SBD and scene analysis tasks. Alignment losses designed specifically for relation queries could also enhance focus and reduce relational ambiguities.

7. Summary and Future Directions

OmniShotCut sets a new standard in SBD by unifying structured relational prediction, deep transformer modeling, large-scale synthetic supervision, and rigorous, domain-diverse benchmarking. This integrated framework balances state-of-the-art localization performance with interpretable relational outputs and offers a pathway toward highly precise video decomposition in unconstrained settings. Future methodologies may expand query structures to capture hierarchies beyond shots (e.g., scenes, acts), further refine synthetic generation strategies, and explore cross-domain relational learning informed by architectures such as the Temporal Perceiver (Tan et al., 2022, Wang et al., 27 Apr 2026).