Anchoring and Rescaling Attention for Semantically Coherent Inbetweening

Abstract: Generative inbetweening (GI) seeks to synthesize realistic intermediate frames between the first and last keyframes beyond mere interpolation. As sequences become sparser and motions larger, previous GI models struggle with inconsistent frames with unstable pacing and semantic misalignment. Since GI involves fixed endpoints and numerous plausible paths, this task requires additional guidance gained from the keyframes and text to specify the intended path. Thus, we give semantic and temporal guidance from the keyframes and text onto each intermediate frame through Keyframe-anchored Attention Bias. We also better enforce frame consistency with Rescaled Temporal RoPE, which allows self-attention to attend to keyframes more faithfully. TGI-Bench, the first benchmark specifically designed for text-conditioned GI evaluation, enables challenge-targeted evaluation to analyze GI models. Without additional training, our method achieves state-of-the-art frame consistency, semantic fidelity, and pace stability for both short and long sequences across diverse challenges.

• The paper introduces Keyframe-anchored Attention Bias and Rescaled Temporal RoPE, two training-free modifications that improve semantic fidelity, frame consistency, and motion pacing in long video inbetweening sequences.
• The method outperforms competing approaches on most reported metrics, reaching 18.17 dB PSNR, 0.6269 SSIM, and 0.2458 FVD on 81-frame clips, while human evaluators rate consistency, semantic fidelity, and pacing substantially higher than Wan’s baseline.
• The authors release TGI-Bench, a 220-video text-annotated benchmark spanning dynamic motion, linear motion, occlusion, and near-static scenes, showing that occlusion and dynamic motion remain the most informative and difficult challenges.

This paper addresses generative inbetweening (GI)—the synthesis of plausible intermediate frames between two widely spaced keyframes—by introducing two training-free attention modifications to a video Diffusion Transformer (DiT), along with the first benchmark dedicated to text-conditioned GI (2603.17651). The work is motivated by three failure modes that intensify as keyframes become sparser and motions more dynamic: degraded semantic fidelity, frame inconsistency, and pace instability.

Background and problem setting

Classical video frame interpolation (VFI) assumes dense keyframes and near-linear motion, and prior diffusion-based GI approaches built on Stable Video Diffusion (SVD), such as TRF and ViBiDSampler, must run the model twice and fuse outputs because SVD accepts only a single conditioning image; this produces collapse and blur on long sequences. DiT-based models such as Wan's First-Last-Frame-to-Video (FLF2V) pipeline can jointly condition on both keyframes and text, but the paper observes that guidance from these conditions weakens for intermediate frames as sequences lengthen. The authors build on Wan2.1-FLF2V-14B-720P, where keyframe latents are concatenated with binary masks into the noisy latent sequence, while UMT5-encoded text and CLIP features of both keyframes form the cross-attention context.

Keyframe-anchored Attention Bias

The first contribution, Keyframe-anchored Attention Bias (KAB), operates in cross-attention. The model's own attention maps are sliced at the rows corresponding to the first and last latent frames, averaged over heads and queries to produce two keyframe anchors $\bar{A}^{(0)}$ and $\bar{A}^{(f-1)}$, then linearly interpolated by temporal position $\tau^{(t)} = t/(f-1)$ to yield frame-wise target anchors $M^{(t)}$. A small logit bias

$$B^{(t)} = \log(M^{(t)}+\varepsilon) - \log(\bar A^{(t)}+\varepsilon)$$

is broadcast over all heads and queries of frame $t$, softly steering intermediate frames toward the interpolated semantic-temporal allocation without destroying local attention structure. Guidance is gated to layers 5–12 during only the first 40% of diffusion steps, with a cosine taper on the strength $\beta^{(t)}$ from 0.7 near the keyframes down to 0.3 at the midpoint, so early layers and steps settle global layout while late stages form texture. Additionally, the baseline FLF2V pipeline fuses $I_{\text{last}}$ and text jointly in one cross-attention while treating $I_{\text{first}}$ separately—an asymmetric design the authors replace with three symmetric cross-attentions ($I_{\text{first}}\leftrightarrow$video, $\bar{A}^{(f-1)}$0video, text$\bar{A}^{(f-1)}$1video), each refined by its own anchor bias and equally weighted.

Rescaled Temporal RoPE

The second contribution, ReTRo, targets frame consistency through self-attention positional encoding. Vanilla RoPE provides relative temporal distances but no explicit anchoring of the two keyframes. ReTRo applies a piecewise scale to the temporal frequency row: $\bar{A}^{(f-1)}$2 on the outer $\bar{A}^{(f-1)}$3 frames per side and $\bar{A}^{(f-1)}$4 in the interior, with height/width frequencies unchanged. Higher edge scales sharpen local attention and preserve keyframe fidelity, while lower mid scales broaden the temporal receptive field of intermediate frames, improving coherence across the whole clip. With defaults $\bar{A}^{(f-1)}$5, this is an architectural-change-free, training-free adjustment applicable to any video DiT.

TGI-Bench

Because existing VFI benchmarks lack natural-language annotations and challenge diversity, the authors curate TGI-Bench: 220 videos from DAVIS, Pexels, and Pixabay, each with ground-truth intermediates, a GPT-4.1-generated caption (manually verified), and one of four challenge labels—dynamic motion, linear motion, occlusion, near-static. Four sequence-length variants (25, 33, 65, 81 frames) enable apples-to-apples comparison across horizons.

Results

Evaluation covers standard metrics (PSNR, SSIM, LPIPS, FID, FVD, VBench) plus GI-specific measures: X-CLIP and a six-model ensemble VQA score for semantic fidelity, LPIPS-frames/CLIPSIM-frames for consistency, and human ratings (>20 participants, double-blind, five-point Likert). Against TRF, ViBiDSampler, GI, FCVG, and Wan, the method achieves the best results on essentially all metrics. On 81-frame sequences it reaches PSNR 18.17 dB, SSIM 0.6269, FID 77.59, FVD 0.2458, and VBench 10.022, versus Wan's 17.63 dB / 82.90 / 9.904. Human evaluation shows the largest margins: 4.38 (frame consistency), 4.27 (semantic fidelity), and 4.34 (pace stability) versus Wan's 3.50 / 3.69 / 3.65—a substantial gap given the five-point scale. Notably, FCVG, despite having no text input, rivals text-conditioned baselines via its motion guidance, which the authors attribute to reduced path ambiguity.

The ablation indicates complementary roles: ReTRo alone yields the second-best overall scores and dominates distortion/distributional metrics (PSNR 18.18, FVD 0.2510), consistent with its role in frame consistency, whereas KAB drives semantic fidelity and pacing gains, and their combination attains the best VBench (10.022) and best or near-best scores on nearly all metrics. Hyperparameter sweeps show moderate sensitivity: KAB with too weak or overly strong guidance degrades quality, and ReTRo with $\bar{A}^{(f-1)}$6 introduces visible artifacts while very low $\bar{A}^{(f-1)}$7 slows apparent motion.

Challenge analysis using TGI-Bench reveals a clear difficulty spectrum: most methods handle near-static cases reliably, while occlusion is hardest for all models. The proposed method achieves the best LPIPS and VBench scores precisely on occlusion and dynamic motion, and the authors observe that performance on these hard subsets accounts for most variation in human preference, making them more informative indicators than near-static cases.

Limitations and open questions

The method is bounded by the generative capacity of the underlying base model. When Wan already produces severely distorted motion or geometry—as in the breakdancing example—the training-free plug-in can only partially correct artifacts, leaving residual unnatural motion. Additional open points include whether framewise temporal rescaling of self-attention RoPE transfers beyond Wan-style DiTs, since the paper demonstrates the approach on a single backbone, and how the linear interpolation assumption underlying target anchors behaves under highly nonlinear trajectories where a straight-line blend of keyframe semantics may be a poor proxy for the true path.

Conclusion

The paper contributes two complementary, training-free attention interventions—cross-attention logit biasing toward interpolated keyframe anchors, and piecewise rescaling of temporal RoPE—together achieving state-of-the-art frame consistency, semantic fidelity, and pace stability for text-conditioned GI, with particularly large human-evaluated margins. TGI-Bench supplies the field's first challenge-stratified, text-annotated evaluation suite for GI, exposing occlusion and dynamic motion as the dominant open failure modes.