Semantic Frame Interpolation (SFI)

• Semantic Frame Interpolation (SFI) is a video generation method that synthesizes intermediate frames using semantic constraints, text prompts, and fixed boundary frames.
• It extends traditional frame interpolation by supporting variable-length outputs and leveraging multimodal inputs, including text and audio, for enhanced temporal coherence.
• Innovative techniques like Mixture-of-LoRA and decoupled cross-attention are employed to balance semantic fidelity with visual continuity, as validated on benchmarks like SFI-300K.

Searching arXiv for the cited papers to ground the article and confirm the referenced works. {"queries":[{"q":"id:(Mathur et al., 2020)"},{"q":"id:(Hong et al., 7 Jul 2025)"},{"q":"id:(Deng et al., 3 Dec 2025)"}]} Semantic Frame Interpolation (SFI) is a generalized conditional video generation problem in which intermediate frames are synthesized between a given first frame and last frame under semantic constraints, rather than being determined solely by low-level motion or appearance continuity. In the formal task definition introduced in "Semantic Frame Interpolation" (Hong et al., 7 Jul 2025), the inputs are a first frame $I_f \in \mathbb{R}^{H \times W \times C}$, a last frame $I_l \in \mathbb{R}^{H \times W \times C}$, a text prompt $T$, and a target number of intermediate frames $N \in \mathbb{Z}^+$, and the output is a sequence $\{I_1, I_2, ..., I_N\}$ that is temporally and visually consistent with the endpoints, follows the semantics specified in $T$, and adapts to the requested sequence length $N$ (Hong et al., 7 Jul 2025). Earlier work on speech videos did not use the term “SFI” explicitly, but "LIFI: Towards Linguistically Informed Frame Interpolation" (Mathur et al., 2020) already articulated a closely related regime in which frame interpolation is constrained by linguistic content, especially visemic and word-level structure. Later multimodal work such as "Beyond Boundary Frames: Audio-Visual Semantic Guidance for Context-Aware Video Interpolation" (Deng et al., 3 Dec 2025) made the semantic framing explicit by guiding interpolation with text, audio, images, and video, thereby treating semantic and contextual signals as central disambiguating constraints rather than auxiliary cues.

1. Definition and scope

The defining formulation of SFI is a deterministic mapping

$$\{I_1, I_2, ..., I_N\} = \mathcal{F}(I_f, I_l, T, N),$$

together with the conditional probabilistic form

$$\{I_1, I_2, ..., I_N\} \sim p(I_{1:N}|I_f, I_l, T, N),$$

or, more explicitly, a model $p_\theta(I_{1:N}|I_f, I_l, T, N)$ whose optimal output is written as

$I_l \in \mathbb{R}^{H \times W \times C}$0

This formulation distinguishes SFI from conventional Video Frame Interpolation (VFI), which usually targets adjacent or near-adjacent frames, interpolates only a few frames, omits text control, and focuses on low-level motion and local consistency (Hong et al., 7 Jul 2025).

A central feature of SFI is variable-length generation. The task explicitly supports arbitrary $I_l \in \mathbb{R}^{H \times W \times C}$1, with the SemFi work targeting 5–81 frames during training while stating that inference can go further (Hong et al., 7 Jul 2025). This enlarges the problem from frame-rate upconversion or dropped-frame recovery to long-range transition synthesis under semantic control. The endpoints $I_l \in \mathbb{R}^{H \times W \times C}$2 and $I_l \in \mathbb{R}^{H \times W \times C}$3 may be semantically very different, and the text prompt may specify creative transitions, style changes, or content insertion and removal (Hong et al., 7 Jul 2025).

This definition also subsumes earlier settings. When $I_l \in \mathbb{R}^{H \times W \times C}$4 is small and $I_l \in \mathbb{R}^{H \times W \times C}$5, SFI reduces to classical interpolation. When $I_l \in \mathbb{R}^{H \times W \times C}$6 is large and text controls the transition, it aligns with foundation video model frame-to-frame generation (Hong et al., 7 Jul 2025). A plausible implication is that SFI functions as a unifying abstraction linking classical motion-centric interpolation, temporally constrained video generation, and text-guided transition synthesis.

2. Precursors in linguistically informed interpolation

Although the term “Semantic Frame Interpolation” was formalized later, LiFi established an important precursor in speech-video interpolation by arguing that semantically faithful interpolation in talking-head videos must preserve correct visemes, maintain proper temporal alignment, and respect word boundaries, prefixes/suffixes, and part-of-speech–driven patterns (Mathur et al., 2020). In that setting, the semantic content is linguistic: what is being said, when it is being said, and how that articulation should appear on the face.

LiFi framed interpolation as reconstruction from corrupted sequences. A network $I_l \in \mathbb{R}^{H \times W \times C}$7 takes a corrupted video $I_l \in \mathbb{R}^{H \times W \times C}$8 and outputs a reconstructed video $I_l \in \mathbb{R}^{H \times W \times C}$9, trained with the frame-level loss

$T$0

The paper’s critique was that standard visual metrics such as MSE, PSNR, and SSIM can remain high even when the resulting lip movements are linguistically incorrect (Mathur et al., 2020). It illustrated this through a synthetic 32-frame sequence formed by repeating the first frame 16 times and the last frame 16 times, and through a Super SloMo example on the sentence “I don’t exactly walk around with a hundred and thirty five million dollars in my wallet,” where the interpolated mouth never fully closes although the real speaker repeatedly closes it; the reported frame metrics were MSE $T$1, PSNR $T$2, and SSIM $T$3 (Mathur et al., 2020).

LiFi therefore advanced a linguistically informed notion of interpolation in which semantically meaningful regions and units matter more than uniform pixel fidelity. Its main architectural contribution, FCN3D + ROI, augmented a 3D fully convolutional denoising autoencoder with a mouth-region objective. The ground-truth mouth ROI $T$4 was extracted using BlazeFace landmarks, and a secondary stream with two convolutional layers and a Spatial Transformer Network learned to crop the mouth region from reconstructed frames. The ROI loss was

$T$5

yielding the total loss

$T$6

The model did not use explicit phoneme labels or ASR-based losses; instead, it used the mouth region as a proxy for visemes (Mathur et al., 2020).

The LiFi evaluation protocol was also semantically structured. Rather than introducing a single new numerical score, it defined six corruption regimes—RandomFrame / Extreme Sparsity, Visemic Corruption, InterWord and IntraWord Corruptions, Prefix Completion, Suffix Completion, and POS-Tagged Corruptions—each designed to probe different dimensions of visual speech understanding (Mathur et al., 2020). This suggests an early form of SFI in which semantics are encoded not only in model design but also in task construction.

3. Formal task structure and relation to adjacent problems

SFI is distinct from, but closely related to, several established problem classes. Relative to classic VFI, SFI extends the input space from local frame pairs to potentially semantically distant endpoints and introduces text as an explicit control variable (Hong et al., 7 Jul 2025). Relative to text-to-video generation, SFI imposes two strong visual boundary conditions, making the problem one of text-guided interpolation with fixed endpoints rather than unconstrained generation (Hong et al., 7 Jul 2025). Relative to video prediction, SFI predicts an interval bounded by known future context rather than extrapolating from the past alone. Relative to video inpainting, SFI can be understood as temporal inpainting with strict boundary conditions and semantic guidance (Hong et al., 7 Jul 2025).

The speech-video interpretation in LiFi sharpens this distinction further. In generic interpolation, a plausible sequence may be accepted if it is visually smooth. In linguistically informed interpolation, smoothness is insufficient if the viseme sequence is wrong, if a lip closure associated with /p/, /b/, or /m/ is omitted, or if a word-internal articulation is temporally misplaced (Mathur et al., 2020). This marks a shift from motion plausibility to semantic fidelity.

The multimodal account in BBF generalizes the same principle beyond speech-only visual conditioning. BBF defines a semantic, context-aware interpolation method in which intermediate frames are guided by multiple modalities—text, audio, images, and video—and are constrained not only by start–end boundary frames but also by the semantics of actions, speech content and rhythm, identity, appearance, and surrounding context (Deng et al., 3 Dec 2025). Here semantics resolve ambiguity in the space of admissible trajectories between endpoints. A plausible implication is that SFI should be viewed less as a narrow variant of interpolation than as a conditional generation problem whose central difficulty is underdetermination of the in-between sequence.

4. Model architectures and conditioning mechanisms

The formal SemFi model proposed specifically for SFI builds on Wan 2.1 I2V, a large diffusion transformer video model operating in latent space with a VAE encoder–decoder (Hong et al., 7 Jul 2025). Wan’s original image-to-video mechanism was extended to dual-frame SFI in three ways. First, a dual guidance tensor was constructed with $T$7 at the first time step, $T$8 at the last time step, and zeros in between. Second, a binary mask $T$9 was set with $N \in \mathbb{Z}^+$0 and $N \in \mathbb{Z}^+$1, marking conditioned versus generated positions. Third, CLIP image embeddings for both $N \in \mathbb{Z}^+$2 and $N \in \mathbb{Z}^+$3 were computed, summed, projected, and injected via cross-attention into the DiT backbone (Hong et al., 7 Jul 2025).

SemFi’s distinctive architectural contribution is Mixture-of-LoRA (MoL). The model uses one universal LoRA $N \in \mathbb{Z}^+$4, active for all frame counts, together with six frame-count-specific expert LoRAs $N \in \mathbb{Z}^+$5 for

$N \in \mathbb{Z}^+$6

At training and inference time, for a requested frame count $N \in \mathbb{Z}^+$7, the applied adaptation is

$N \in \mathbb{Z}^+$8

Each LoRA component uses rank 16, the base Wan parameters are frozen, and only the LoRA parameters are trained (Hong et al., 7 Jul 2025). The stated motivation is that a single low-rank adaptation cannot simultaneously specialize in short-range interpolation, medium-range transition, and long-range semantic generation.

BBF adopts a different but complementary architecture. It is a DiT-based latent video diffusion model built on the Wan2.1-I2V backbone, with start and end frames encoded as hard constraints at the temporal boundaries of the latent sequence $N \in \mathbb{Z}^+$9 (Deng et al., 3 Dec 2025). It introduces a decoupled multimodal cross-attention pipeline in which text is injected first to establish object, scene, action, and intent semantics, followed by a shared image+audio branch that refines appearance and temporal dynamics (Deng et al., 3 Dec 2025). The audio stream is produced by encoding the audio track with Wav2Vec 2.0 to obtain $\{I_1, I_2, ..., I_N\}$0, then refining those features with an audio adapter trained using the start–end latent difference

$\{I_1, I_2, ..., I_N\}$1

yielding $\{I_1, I_2, ..., I_N\}$2 that carries both audio content and motion progression information (Deng et al., 3 Dec 2025).

LiFi’s FCN3D + ROI and BBF’s region-focused diffusion training share a notable design intuition despite operating in different regimes. In BBF, face and lip masks $\{I_1, I_2, ..., I_N\}$3 and $\{I_1, I_2, ..., I_N\}$4 are extracted via MediaPipe and combined into the continuous weight map

$\{I_1, I_2, ..., I_N\}$5

with the reconstruction loss

$\{I_1, I_2, ..., I_N\}$6

Early training emphasizes global structure and motion, whereas later training increases facial and lip weighting to refine semantically critical details for lip-speech synchronization (Deng et al., 3 Dec 2025). This parallels LiFi’s mouth-region bias, but in a multimodal latent diffusion setting.

5. Datasets, benchmarks, and evaluation methodology

The formal SFI benchmark introduced with SemFi is SFI-300K, described as the first general-purpose dataset and benchmark specifically designed for SFI (Hong et al., 7 Jul 2025). From Table 1 in that work, the dataset contains 300,000 clips, 5833 total minutes, source resolutions including 1920×1080, 1080×1920, and 2048×1080, frame lengths from 5 to 81, and high-density text captions (Hong et al., 7 Jul 2025). It is built upon the Open Sora Plan datasets. Videos are filtered to FPS $\{I_1, I_2, ..., I_N\}$7 and total frame count between $\{I_1, I_2, ..., I_N\}$8 and $\{I_1, I_2, ..., I_N\}$9, with

$T$0

First and last frames are compared using a CLIP score $T$1, defined as cosine similarity between image embeddings, and a flow score $T$2, defined as average optical flow magnitude computed with RAFT. Videos outside threshold ranges are removed to retain meaningful but not chaotic start–end changes (Hong et al., 7 Jul 2025). Multi-frame clips are then cut around the temporal midpoint for each $T$3 according to

$T$4

and long-text captions are generated using Qwen2.5-VL-32B (Hong et al., 7 Jul 2025).

SFIBench is the associated test subset. It consists of 100 randomly selected videos from SFI-300K, each evaluated at all six frame scales, producing 600 test clips with endpoints, captions, and ground-truth intermediate frames (Hong et al., 7 Jul 2025). Metrics span several groups: Video Fidelity via LPIPS and FID; Frame Fidelity via PSNR, SSIM, and LPIPS on generated first and last frames relative to the input condition images; Semantic Fidelity via ViCLIP video–text feature similarity; and VBench-derived Video Quality metrics including Temporal Flickering (TF), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), and Imaging Quality (IQ) (Hong et al., 7 Jul 2025).

LiFi, by contrast, builds on LRS3-TED, reporting approximately 118,516 utterances, 5091 speakers, approximately 3.9M word instances, and a vocabulary of approximately 51k words, with frame rates of 24, 30, and 60 fps unified to 32 fps (Mathur et al., 2020). It uses the LRS3 “Pretrain” split for training and the “trainval” split both to define corruption-based evaluation sets and to serve as the reconstruction test set (Mathur et al., 2020). The released challenge datasets align with the six corruption types; for example, RandomFrame & Extreme Sparsity use 4,004 speakers, 31,982 utterances, 356,940 word instances, and vocabulary 17,545, while the Visemic set uses 2,883 speakers, 6,152 utterances, 338,207 word instances, and vocabulary 16,663 (Mathur et al., 2020).

BBF evaluates on both generic and semantic settings. Its training set is CelebV-HQ with 35,666 high-resolution face video clips. Evaluation uses DAVIS 2017, HDTF, and Hallo3. Generic VFI metrics on DAVIS and HDTF are FID, FVD, and LPIPS, while audio-driven talking-head evaluation on HDTF and Hallo3 additionally uses PSNR, SSIM, and Sync-D for lip-speech synchronization (Deng et al., 3 Dec 2025).

6. Empirical findings and characteristic failure modes

SemFi’s main quantitative results on SFIBench show a characteristic trade-off between semantic alignment and raw frame fidelity. On aggregated evaluation, Video Fidelity scores are LPIPS 0.2664 and FID 74.52, second to FCVG on both; Frame Fidelity scores are PSNR 21.96, SSIM 0.6576, and endpoint LPIPS 0.2025, behind Wan and FCVG; Semantic Fidelity is 0.2217, the best among compared methods; and Video Quality scores are TF 0.9816, MS 0.9901, DD 0.2333, AQ 0.5114, and IQ 0.6129, with SemFi best on MS and second on several others (Hong et al., 7 Jul 2025). Appendix comparisons show that for small frame counts, particularly 5 and 9, Wan performs very badly relative to SemFi; at 5 frames, Wan has LPIPS 0.4772 and FID 275.1, while SemFi obtains LPIPS 0.2472 and FID 86.63 (Hong et al., 7 Jul 2025). Variance analysis across frame counts further shows that SemFi is much more stable than Wan on LPIPS, FID, Semantic Fidelity, TF, MS, AQ, and IQ (Hong et al., 7 Jul 2025).

The SemFi ablations indicate that multi-frame training is essential. A model trained only on 65-frame data (“w/o Multi-frame”) yields LPIPS 0.2841, FID 109.2, Semantic Fidelity 0.2190, AQ 0.4922, and IQ 0.5961. Multi-frame training without MoL (“w/o MoL”) improves to LPIPS 0.2476, FID 77.04, Semantic Fidelity 0.2208, AQ 0.5100, and IQ 0.6149. The full SemFi model with MoL gives LPIPS 0.2664, FID 74.52, Semantic Fidelity 0.2217, AQ 0.5114, and IQ 0.6129 (Hong et al., 7 Jul 2025). Qualitative analysis attributes MoL’s value to improved temporal allocation of semantic change: without MoL, transitions may compress into a short segment of an 81-frame sequence, leaving the remainder static; with MoL, change is distributed more uniformly (Hong et al., 7 Jul 2025).

LiFi’s findings concentrate on the mismatch between visual metrics and linguistic correctness. In random corruption, FCN3D+ROI substantially improves over FCN3D; for example, at 75% corruption the reported PSNR is 24.75 versus 20.79 (Mathur et al., 2020). In viseme corruption, FCN3D+ROI reaches PSNR 26.54 and SSIM 0.91, surpassing or matching Super SloMo at PSNR 25.26 and SSIM 0.91 (Mathur et al., 2020). In word-level and multi-word corruption, FCN3D+ROI also shows large gains, with word-level PSNR 25.33 versus 18.46 for FCN3D and multi-word PSNR 23.14 versus 19.61 (Mathur et al., 2020). Prefix and suffix tasks favor BDLSTM because of stronger sequential modeling, though the ROI mechanism narrows the gap (Mathur et al., 2020). The characteristic failure mode of models trained only with per-pixel losses is smoothed or averaged mouth shapes and missed articulatory closures; optical-flow models such as Super SloMo can produce sharp images while remaining semantically wrong (Mathur et al., 2020).

BBF’s experiments support the value of multimodal semantic guidance. In generic interpolation, BBF reports on DAVIS FID 262.30, FVD 1034.41, and LPIPS 0.54, and on HDTF FID 10.50, FVD 174.24, and LPIPS 0.12, described as generally best FID and FVD on DAVIS and best FVD on HDTF (Deng et al., 3 Dec 2025). In audio-visual synchronized interpolation, BBF on HDTF achieves FID 25.01, FVD 297.70, LPIPS 0.18, PSNR 20.44, SSIM 0.72, and Sync-D 1.90; on Hallo3 it obtains FID 31.95, FVD 773.00, LPIPS 0.52, PSNR 10.86, SSIM 0.44, and Sync-D 1.42 (Deng et al., 3 Dec 2025). The paper further states that on Hallo3 BBF outperforms StableAvatar by 9.3% in FID, 38.8% in FVD, and improves PSNR, SSIM, and Sync-D from 4.50 to 1.42 (Deng et al., 3 Dec 2025). Ablations on input conditions show that text only yields the best FID, PSNR, and SSIM but the worst Sync-D; audio only improves Sync-D but degrades FID, PSNR, and SSIM; text + audio achieves the best Sync-D and FVD with competitive other metrics (Deng et al., 3 Dec 2025). This directly indicates complementarity of semantic modalities.

7. Conceptual significance, limitations, and future directions

SFI reorients interpolation around semantic disambiguation. In flow-based formulations, the interpolated frame is typically produced by warping the endpoints with estimated flows,

$T$5

under losses that promote endpoint fidelity and flow smoothness (Deng et al., 3 Dec 2025). BBF formalizes the tendency of such methods toward low-curvature, approximately constant-velocity trajectories through the trajectory energy

$T$6

with first-order optimality yielding $T$7 and therefore

$T$8

The stated implication is that optical-flow-based interpolation tends to smooth out rapid, semantically important micro-motions such as phoneme transitions (Deng et al., 3 Dec 2025). SFI, by contrast, uses semantics to select trajectories consistent with high-level events and contextual signals.

Across the three works, several limitations recur. LiFi has no direct semantic or phoneme-level loss, no explicit audio or text conditioning, a frontal talking-head bias inherited from LRS3, and evaluation that ultimately still uses SSIM and PSNR rather than WER, CER, or viseme accuracy (Mathur et al., 2020). SemFi is trained only up to about 81 frames, uses a predominantly real-world dataset with very little synthetic or stylized content, and therefore may exhibit a domain gap on cartoons, CGI, and 3D renderings (Hong et al., 7 Jul 2025). BBF depends on reliable face and lip masking, is trained mainly on celebrity-face video, requires multiple heavy encoders such as Qwen3-VL, CLIP, and Wav2Vec, and does not always achieve the absolute best synchronization metric against specialized lip-sync models (Deng et al., 3 Dec 2025).

The future directions named in these works point toward a richer SFI agenda. LiFi explicitly states: “In the future, we would like to cover more such linguistic aspects of speech. A parallel task would also be to take the audio modality into account while reconstructing the corrupted and missing frames in video interpolation” (Mathur et al., 2020). Its discussion also proposes audio-conditioned interpolation, lip-reading or ASR-based semantic losses, and textual semantics as natural extensions (Mathur et al., 2020). SemFi proposes architectural upgrades to MoL for longer sequences, more granular or continuous routing over frame count, and expansion of the dataset with synthetic videos (Hong et al., 7 Jul 2025). BBF suggests richer semantic modeling including emotion, intent, prosody, and sentiment, multilingual scenarios, extension beyond faces to full-body motion and multi-person interactions, and tighter theoretical understanding of how semantic guidance reshapes the diffusion posterior for interpolation (Deng et al., 3 Dec 2025).

Taken together, these works establish SFI as a broad research program rather than a single architecture. LiFi demonstrates that semantically wrong interpolation can score well on conventional pixel metrics in speech video, thereby motivating semantically structured objectives and benchmarks (Mathur et al., 2020). SemFi formalizes SFI as a unified task with dual-frame constraints, text control, arbitrary frame count, and a dedicated benchmark (Hong et al., 7 Jul 2025). BBF extends the paradigm to multimodal semantic guidance with diffusion transformers, decoupled cross-modal fusion, and audio-visual synchronization (Deng et al., 3 Dec 2025). The resulting picture is of interpolation not as mere estimation of missing frames, but as conditional generation of temporally coherent trajectories constrained by meaning.