OmniShotCutBench: SBD Benchmark Suite

• OmniShotCutBench is a comprehensive benchmark suite for shot boundary detection, covering diverse editing styles and modern video domains.
• It provides high-quality, per-frame annotations and advanced metrics like Transition IoU and sudden-jump accuracy for nuanced performance evaluation.
• The benchmark incorporates a synthetic transition synthesis pipeline with 11.9 million labeled transitions, enhancing training robustness across varied editing phenomena.

OmniShotCutBench is a comprehensive benchmark suite designed for wide-domain, fine-grained evaluation of shot boundary detection (SBD) systems in modern video editing contexts. Introduced in the context of OmniShotCut, a Transformer-based SBD method, OmniShotCutBench addresses limitations of prior benchmarks—including outdated content domains, annotation ambiguity, limited transition taxonomies, and coarse evaluation protocols—by providing high-quality, per-frame annotations and rigorous, diagnostic metrics applicable across diverse video genres and sophisticated transition types (Wang et al., 27 Apr 2026).

1. Dataset Composition and Domain Coverage

OmniShotCutBench consists of 114 video clips, each standardized to 30 FPS and 480p resolution, with a total duration of approximately 110 minutes. All clips are truncated to 60 seconds or less. On average, each 1-minute video contains 5–10 annotated shot boundaries, totaling approximately 600–1,200 boundaries, split evenly between instantaneous (hard-cut or sudden-jump) and gradual (dissolve, wipe, fade, etc.) transitions.

The sampling strategy intentionally covers a wide range of contemporary editing styles beyond legacy broadcast domains. The included domains are:

• Vlogs (lifestyle and tutorials)
• Short-form social media
• Anime
• Movies and concert footage
• Documentaries and news
• Gameplay and screen-capture recordings
• Sports
• Urban and travel footage
• Unboxing/product reviews
• Mixed screen-recorded content (e.g., software demos)

This broad coverage reflects the heterogeneity of modern internet video and exposes models to diverse editing cues, semantic shifts, and transition effects not represented in older SBD benchmarks.

2. Synthetic Transition Synthesis Pipeline

A key innovation in OmniShotCutBench is the accompanying synthetic transition pool used for training. The pipeline synthesizes transitions from a library of 1.5 million clean video clips, producing 11.9 million labeled transitions. The synthetic data generator encapsulates:

A. Clip-level composition: The number of clips per synthetic video is sampled from a Poisson distribution $k \sim \mathrm{Pois}(\lambda=7.0)$, clipped to $k \in [1,28]$. For multi-clip videos, durations follow $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$; for single-clip videos (no transition), $d \sim \mathcal{N}(8.0\,\mathrm{s},\,1.0^2)$. Seventy-five percent of successive clips are selected to maximize semantic coherence (same DINOv3 cluster), with the remainder sampled cross-cluster to increase scene diversity. Twenty-five percent of synthetic videos use an "extreme dense mode" (28 clips, durations uniform in $[0.15,1.0]$ s).

B. Transition families and ratios: Nine major transition types and $\simeq 30$ subtypes are represented, with empirical mixing as follows:

Transition Family	Typical Ratio	Subtypes
Hard Cut	35%	Instantaneous transition
Dissolve (total 13.6%)	9.4/2.4/1.8%	Transparent, Cross-blur, Ripple
Wipe (total 14.1%)	Various	Directional, Diagonal, Circular, Mosaic, etc.
Push	≃6.5%	Puzzle push
Slide	≃10.6%	Whip-pan, Cube
Zoom	≃10.2%	In/out, Spin, Cross-zoom, Swap zoom
Fade	≃8.7%	To/from black/white, Dip
Doorway open	2.9%

C. Parametric control: Gradual blends are rendered via $\alpha(t) = t/T$, where $t$ indexes the frame and $T$ is the transition duration, with $$T \sim \mathrm{Uniform}(\lceil0.15s \cdot \mathrm{FPS}\rceil, \lfloor2.5s \cdot \mathrm{FPS}\rfloor)$$. Whip-pan slides use a constrained $k \in [1,28]$0. Very short transitions ($k \in [1,28]$1 frames) default to hard cuts. Sudden-jump augmentations are applied to 90% of hard cuts by randomly cropping segments $k \in [1,28]$2 frames. Additional parameters govern motion masks, blur, feather, color wash, lighting, and overlay effects.

A plausible implication is that the broad and parameterized transition family in the synthetic pool allows for training models sensitive to both subtle and compounded editing phenomena.

3. Annotation Protocol and Taxonomy

Annotation rigor is ensured through a protocol involving:

• Pre-labeling study with professional video-editing tutorials and explicit boundary-definition guidelines.
• Multiple annotation pilots on held-out data for consistency alignment.
• A frame-level labeling tool supporting timeline-based, frame-precise insertion, open-image inspection, multi-selection, and auto-saves.

For each annotated boundary, annotators record:

• Start and end frame indices of the transition region
• One of eight intra-shot classes: General/Vanilla, Dissolve, Wipe, Push, Slide, Zoom, Fade, Doorway
• Inter-shot relation: Transition, Hard-Cut, Sudden-Jump, or “New-Start”
• Confidence score $k \in [1,28]$3 reflecting perceptual ambiguity

Ambiguous cases are adjudicated via lead annotator review and resolved through consensus.

4. Evaluation Metrics, Diagnostics, and Analysis

OmniShotCutBench introduces comprehensive evaluation and diagnostic metrics extending beyond conventional SBD evaluation:

4.1 Traditional Range Metrics

Using a 2-frame tolerance $k \in [1,28]$4:

• True positives: boundary-pair matches where $k \in [1,28]$5 and $k \in [1,28]$6
• Precision: $k \in [1,28]$7
• Recall: $k \in [1,28]$8
• F1: $k \in [1,28]$9

4.2 Transition IoU

For gradual transitions, the intersection-over-union is defined as:

$d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$0

Averages are optionally weighted by $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$1.

4.3 Sudden-Jump Accuracy

Detection requires exact ($d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$2) matching for “Sudden-Jump” boundaries:

$d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$3

4.4 Relational Classification Accuracy

After best-match assignment via IoU:

• $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$4
• $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$5

4.5 Additional Diagnostics

• Confidence-weighted F1 as a function of $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$6
• IoU histograms for transition family analysis (“bleeding” diagnosis)
• Confusion matrices for intra/inter labels to pinpoint systematic classification errors

These diagnostics allow researchers to stratify model performance by transition type, ambiguity, and relational context.

5. Comparative Assessment with Prior Benchmarks

Earlier SBD benchmarks (BBC, RAI, IACC3, TRECVID) are limited to broadcast news and documentaries, with coarse labels (mainly hard cuts, few dissolves), low domain diversity, vague definitions, no per-boundary confidence, and only range-based metrics with high tolerance (5–10 frames). Relational and IoU-based diagnostics are absent.

In contrast, OmniShotCutBench offers:

• Wide-domain, internet-aligned video sources
• Per-frame, pilot-aligned annotations with 2-frame tolerance and confidence scores
• Eight-way intra–shot and three-way inter–shot relational taxonomy (inc. “sudden jump”)
• Transition IoU and zero-tolerance sudden-jump accuracy metrics
• Diagnostic metrics exposing specific failure modes not visible under legacy protocols
• High annotation consistency from consensus-reviewed, tool-supported protocols

A plausible implication is that these enhancements enable both more sensitive model evaluation and diagnosis of previously concealed weaknesses in SBD systems.

6. Performance Results and Benchmark Impact

Reported results on OmniShotCutBench show substantial performance gaps between OmniShotCut and prior SBD systems:

Method	Trans-IoU	SJ-Acc	P	R	F1
PySceneDetect [29]	0.183	0.416	0.833	0.689	0.754
TransNet V2 [60]	0.192	0.261	0.913	0.734	0.814
AutoShot [61]	0.252	0.455	0.849	0.782	0.814
OmniShotCut	0.632	0.761	0.898	0.858	0.883

Notable performance deltas include a +0.38 uplift in Transition IoU over prior best, near doubling of sudden-jump accuracy compared to TransNet V2, and a range F1 improvement of ≃0.07. Additionally, relational classification is newly quantified: $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$7, $d \sim \mathcal{N}(2.8\,\mathrm{s},\,1.6^2)$8.

This suggests that the benchmark substantially elevates the standard for SBD evaluation, particularly regarding nuanced transitions and relational understanding. The introduction of confidence-annotation, per-family diagnostics, and relational metrics positions OmniShotCutBench as a diagnostic tool as well as a leaderboard, surfacing failure modes masked by prior, coarser benchmarks (Wang et al., 27 Apr 2026).