SFI-300K: Multiscale Dataset for Semantic Frame Interpolation

• SFI-300K is a large-scale, high-quality multimodal dataset with 300K video clips and detailed captions designed specifically for semantic frame interpolation.
• It formulates the task as a conditional generation problem where models combine endpoint images, text prompts, and variable frame counts to generate coherent intermediate frames.
• The dataset underpins the SFIBench benchmark and the SemFi model, demonstrating superior scale-adaptive interpolation compared to traditional video frame interpolation methods.

SFI-300K is a high-quality, large-scale multimodal dataset with 300,000 video clips and rich, high-density captions, introduced as the first general-purpose dataset and benchmark specifically designed for Semantic Frame Interpolation (SFI) (Hong et al., 7 Jul 2025). In the SFI formulation, a model receives a first frame $I_f$, a last frame $I_l$, a text prompt $T$, and a desired intermediate frame count $N$, and must generate a sequence of intermediate frames that is semantically coherent with the text and visually continuous with the boundary frames. The dataset was created because conventional video frame interpolation datasets and newer general-purpose video datasets do not fully match the paired-endpoint, multi-frame, text-controlled interpolation setting required by SFI.

1. Task definition and conceptual scope

SFI-300K is organized around the task of Semantic Frame Interpolation, defined as a conditional generation problem rather than a narrowly geometric inbetweening problem (Hong et al., 7 Jul 2025). The paper formalizes the target as

$$\{I_1, I_2, ..., I_N\} = \mathcal{F}(I_f, I_l, T, N),$$

with the corresponding probabilistic form

$$\{I_1, I_2, ..., I_N\} \sim p(I_{1:N}\mid I_f, I_l, T, N).$$

This formulation makes the task explicitly multimodal: endpoint images constrain boundary conditions, the text prompt constrains semantics, and $N$ controls temporal scale.

The paper places SFI between two pre-existing regimes. When $N$ is small and $T=\emptyset$, the task reduces to traditional VFI. When $N$ is large, it becomes similar to foundation-video-model-based frame-to-frame generation. This positioning is central to the rationale for SFI-300K: conventional VFI datasets are often small, semantically weak, or limited to short clips with only tiny motion changes between endpoints, while large video datasets are general-purpose and not constructed around the paired-endpoint, multi-frame, text-controlled interpolation setting. A common misunderstanding is to treat SFI-300K as merely a larger VFI corpus; the paper instead defines it as infrastructure for semantic, controllable, multi-length interpolation.

2. Dataset composition and statistical profile

SFI-300K is reported as having 300,000 videos/clips, 5833 total minutes, source resolutions including 1920×1080, 1080×1920, 2048×1080, ..., frame counts from 5 to 81, semantic information score listed as 52, and task domain SFI (Hong et al., 7 Jul 2025). Each sample includes rich captions, and the dataset spans diverse content categories. The paper also states that SFI-300K contains predominantly real-world footage, with minimal synthetic content.

Property	Value
Videos/clips	300,000
Total duration	5833 minutes
Frame-length range	5–81
Discrete frame scales	5, 9, 17, 33, 65, 81
Source resolutions	1920×1080, 1080×1920, 2048×1080, ...
Semantic information score	52
Task domain	SFI

The six discrete frame scales are especially important because they instantiate the multi-length setting rather than treating clip duration as a nuisance variable. The paper further notes that CLIP and flow-score distributions are broader than earlier datasets, implying more variation in content and motion between first and last frames. This design aligns the corpus with scenarios in which the boundary frames may be substantially dissimilar, rather than trivially adjacent.

3. Data collection, filtering, and annotation pipeline

The dataset is curated from publicly available datasets used in Open-Sora Plan and is produced through a staged collection-and-processing pipeline (Hong et al., 7 Jul 2025). Videos with FPS $I_l$0 are filtered out, and videos are required to have a total frame count between $I_l$1 and $I_l$2, where $I_l$3. The first and last frames are then extracted from each video.

Two endpoint scores are computed. The first is a CLIP score $I_l$4 using CLIP image features, where higher means the first and last frames are more similar. The second is a flow score $I_l$5 using RAFT, defined as the average optical-flow $I_l$6-norm over all pixels, where higher means greater difference between endpoints. The paper states that these scores are analyzed across all videos, that high and low thresholds are defined for each, and that videos outside the threshold range are removed. The explicit goal is to retain videos with meaningful changes between first and last frames.

After filtering, videos are segmented according to the frame-count set

$I_l$7

using

$I_l$8

Caption annotation is then performed with Qwen2.5-VL-32B using a carefully designed prompt to produce high-perception, high-information-density long-text semantic captions. The final dataset is written as

$I_l$9

where $T$0 is the set of clips and $T$1 is the caption set.

4. SFIBench: benchmark construction and evaluation protocol

The accompanying benchmark, SFIBench, is created by randomly selecting 100 videos from SFI-300K and extracting all 6 frame scales per video, giving 600 video clips total (Hong et al., 7 Jul 2025). Its evaluation protocol is deliberately multidimensional and covers fidelity to ground truth, adherence to conditioning frames, semantic alignment, perceptual quality, and robustness across different interpolation lengths.

The benchmark evaluates four major dimensions. Video Fidelity is measured by LPIPS and FID, computed between all generated frames and their corresponding ground-truth frames. Frame Fidelity is measured by PSNR, SSIM, and LPIPS, computed between the first/last generated frames and their corresponding conditional input images, to assess adherence to the boundary frames. Semantic Fidelity is measured using ViCLIP feature similarity between generated videos and the text prompt. Video Quality adopts five dimensions from VBench: Temporal Flickering (TF), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), and Imaging Quality (IQ).

The definitions of the video-quality metrics are explicit. Temporal Flickering is the mean absolute error between consecutive generated frames. Motion Smoothness reconstructs odd-numbered frames from even-numbered frames using AMT, then computes the MAE between original and reconstructed odd frames. Dynamic Degree is the optical-flow strength between consecutive frames estimated with RAFT. Aesthetic Quality is the average score from the LAION aesthetic predictor on each frame. Imaging Quality is the average of frame-wise MUSIQ scores.

A distinctive feature of SFIBench is that it evaluates across all six lengths—5, 9, 17, 33, 65, and 81—rather than collapsing performance into a single setting. The paper explicitly uses per-frame-count quantitative comparison on each of the six lengths, aggregate “All” results over the whole test set, and $T$2-transformed variance across frame counts to measure stability. This turns multi-scale robustness into a first-class benchmark objective.

5. Baselines, SemFi, and reported benchmark results

SFI-300K functions not only as a training corpus but also as the empirical substrate used to validate SemFi, the paper’s model built on Wan2.1 / Wan-14B-I2V and fine-tuned with Mixture-of-LoRA (MoL) (Hong et al., 7 Jul 2025). In the benchmark experiments, the main baselines are GI, FCVG, Wan, and FILM. GI and FCVG are diffusion-based generative inbetweening methods. Wan is the foundation video model used as SemFi’s backbone. FILM appears only in qualitative comparison because it can only generate $T$3 intermediate frames and cannot handle all test cases.

SemFi uses 1 universal LoRA plus 6 frame-count-specific LoRA experts, with the frame-count set

$T$4

For a target frame count $T$5, it picks the expert minimizing $T$6: $T$7 Training details relevant to benchmark use are also specified: frozen backbone, only MoL trained, 7 LoRA components, rank 16, 1 epoch on SFI-300K excluding SFIBench samples, AdamW, batch size 32, learning rate $T$8, and all videos resized to $T$9.

On the full SFIBench test set, the paper reports the following comparison between SemFi and Wan:

Metric	SemFi	Wan
LPIPS	0.2664	0.3016
FID	74.52	115.0
PSNR	21.96	26.87
SSIM	0.6576	0.7878
Frame LPIPS	0.2025	0.1790
Semantic fidelity	0.2217	0.2204
TF	0.9816	0.9509
MS	0.9901	0.9602
DD	0.2333	0.4017
AQ	0.5114	0.4893
IQ	0.6129	0.6055

The paper’s interpretation is that SemFi is better overall than Wan and GI in semantic fidelity and video quality. It also states that Wan can do well on some fidelity metrics in some settings, but is unstable, especially for low-frame cases; that Wan is particularly poor at low frame counts, especially 5 frames; and that VFI methods degrade as the frame count increases and often collapse into flash/fade-like transitions rather than coherent interpolation. The appendix further reports per-frame-length results for 5, 9, 17, 33, 65, and 81 frames, with SemFi described as more stable across all scales.

6. Ablations, limitations, and research significance

Two ablations are reported in direct connection with SFI-300K’s multi-scale design (Hong et al., 7 Jul 2025). The w/o Multi-Frame variant is trained only on 65-frame data and performs worse in scale adaptation. The w/o MoL variant removes the Mixture-of-LoRA module and shows worse behavior in frame allocation and temporal control. The stated conclusion is that both multi-frame training data and MoL are important for scale-adaptive interpolation.

The paper also states two major limitations. First, the maximum training frame length is about 81. Although inference can generate videos up to hundreds of frames, the training frame-length ceiling limits the supported range and constrains the upper bound of the frame set $N$0. Second, SFI-300K contains minimal synthetic content, creating a gap for synthetic interpolation scenarios. Planned future work includes improving the MoL architecture for long-sequence adaptation, refining the dataset with quality-based filtering, and adding carefully designed synthetic content to bridge the domain gap.

Within the paper’s framing, SFI-300K matters because it supplies the paired endpoints, semantic captions, and multi-length clips needed to train and evaluate semantic, controllable, multi-scale frame interpolation models. Its broader significance lies in standardizing a regime that previously lacked a clear official definition and a well-established benchmark. This suggests that the dataset’s main contribution is not only scale, but the consolidation of task definition, curation procedure, and evaluation protocol into a single reproducible benchmark substrate.