Papers
Topics
Authors
Recent
Search
2000 character limit reached

SwiftI2V: Efficient High-Resolution Image-to-Video Generation via Conditional Segment-wise Generation

Published 7 May 2026 in cs.CV | (2605.06356v1)

Abstract: High-resolution image-to-video (I2V) generation aims to synthesize realistic temporal dynamics while preserving fine-grained appearance details of the input image. At 2K resolution, it becomes extremely challenging, and existing solutions suffer from various weaknesses: 1) end-to-end models are often prohibitively expensive in memory and latency; 2) cascading low-resolution generation with a generic video super-resolution tends to hallucinate details and drift from input-specific local structures, since the super-resolution stage is not explicitly conditioned on the input image. To this end, we propose SwiftI2V, an efficient framework tailored for high-resolution I2V. Following the widely used two-stage design, it addresses the efficiency--fidelity dilemma by first generating a low-resolution motion reference to reduce token costs and ease the modeling burden, then performing a strongly image-conditioned 2K synthesis guided by the motion to recover input-faithful details with controlled overhead. Specifically, to make generation more scalable, SwiftI2V introduces Conditional Segment-wise Generation (CSG) to synthesize videos segment-by-segment with a bounded per-step token budget, and adopts bidirectional contextual interaction within each segment to improve cross-segment coherence and input fidelity. On VBench-I2V at 2K resolution, SwiftI2V achieves performance comparable to end-to-end baselines while reducing total GPU-time by 202x. Particularly, it enables practical 2K I2V generation on a single datacenter GPU (e.g., H800) or consumer GPU (e.g., RTX 4090).

Summary

  • The paper introduces SwiftI2V, a two-stage image-to-video system that combines low-resolution motion generation with image-anchored 2K detail synthesis and conditional segment-wise generation.
  • SwiftI2V achieves a leading VBench-I2V score of 6.4244 at 2560×1408 resolution, including 0.9975 background fidelity and 0.9910 subject fidelity across 81-frame videos.
  • The system reduces GPU time by 202× versus CineScale, runs in about 111 seconds on one H800 or approximately 380 seconds on an RTX 4090 with CPU offloading, and supports segmented streaming output.

Motivation and problem setting

High-resolution image-to-video (I2V) generation requires synthesizing plausible temporal dynamics while preserving the spatial structure and fine-grained textures of a given input image. At 2K resolution, this task is dominated by two coupled difficulties. First, token counts grow rapidly with resolution, making attention-based Diffusion Transformer (DiT) inference expensive in both memory and latency. Second, I2V imposes stricter fidelity requirements than text-to-video (T2V): appearance drift from the input image becomes less tolerable as resolution increases, since high-frequency details such as textures and identity cues must remain stable across frames.

The paper identifies two existing paradigms and their respective failures. End-to-end high-resolution generation with a single model couples global motion modeling and fine-detail synthesis in one objective, which demands large backbones, many sampling steps, and prohibitive GPU resources. The alternative low-resolution generation plus video super-resolution (LR+VSR) pipeline reduces cost but typically applies a generic VSR model that is not explicitly conditioned on the input image, leading to hallucinated details and drift from input-specific local structures. SwiftI2V is designed to occupy the favorable corner of this trade-off: end-to-end-level fidelity at LR+VSR-level cost.

Method

SwiftI2V follows the widely adopted motion–detail decoupled two-stage design, but its contribution lies in how each stage is instantiated for strongly image-conditioned 2K synthesis.

Stage I: low-resolution motion reference. The input image is downsampled to 360P (640×352640\times352), and a large-capacity DiT backbone (Wan2.1-I2V-480P) generates a low-resolution video serving as a motion and structure reference. A rank-128 LoRA adapts the backbone to 360P I2V, and an off-the-shelf few-step distillation LoRA enables 4-step sampling without classifier-free guidance.

Hybrid reference construction. Stage I's output is upsampled to target resolution, and its first frame is replaced by the full-resolution input image. This first-frame replacement injects the input as a boundary condition, reducing first-frame mismatch relative to conventional VSR pipelines while retaining Stage I's motion cues.

Stage II: conditional high-resolution synthesis. Rather than treating refinement as generic super-resolution, Stage II uses a smaller DiT (Wan2.2-TI2V-5B, fully fine-tuned) with a higher-compression 3D VAE (downsampling factors (16,16,4)(16,16,4)). The hybrid reference latent zref\mathbf{z}_{\mathrm{ref}} is channel-concatenated with the noisy latent, whose first temporal block is replaced by the encoded input image zx\mathbf{z}_x, acting as an explicit appearance anchor. The patch-embedding input channels are expanded from 48 to 96, initialized by copying pretrained weights and zero-initializing the new channels. An appendix validates that this VAE preserves high-frequency content at 2K (PSNR 35.25 dB, SSIM 0.941), with power spectral density curves overlapping the input even above normalized frequency 0.6 — supporting the claim that aggressive compression does not sacrifice detail recovery capacity.

Conditional segment-wise generation

The central scalability mechanism is Conditional Segment-wise Generation (CSG). The DiT input is split along time into blocks; the first block is the image anchor, and the remaining t−1t-1 blocks are partitioned into S=⌈(t−1)/M⌉S=\lceil(t-1)/M\rceil segments of MM noisy blocks each. Each diffusion step processes only a window consisting of the anchor block, the last NN finalized neighbor blocks, and the current segment, bounding the per-step 2K token budget. With (M,N)=(3,1)(M,N)=(3,1), the 20 generative blocks of an 81-frame video are covered by seven segments.

Two design choices distinguish CSG from prior streaming approaches such as HiStream, CaSuVid, and MAGI-1:

  • Bidirectional contextual interaction. Within each window, standard attention lets the anchor, neighbor, and noisy blocks attend to each other bidirectionally, rather than imposing a causal mask where history is read-only context. Crucially, updates are applied only to the current segment; conditioning latents are never written back, so finalized segments remain immutable. This avoids autoregressive error accumulation while still allowing context features to adapt dynamically during denoising.
  • Teacher-forced training under CSG. Conditioning blocks are drawn from ground-truth videos during training, with loss computed only on noisy blocks, so the model learns to exploit anchored appearance and short-range context.

A third component addresses the cascade interface gap. Because Stage II is trained on clean downsampled clips but consumes Stage I outputs containing VAE distortion and flicker at inference, the authors synthesize Stage-I-like training inputs by adding noise (σ=0.10\sigma=0.10, one denoising step) to downsampled clips and denoising them with Stage I. These pairs preserve ground-truth motion correspondence while exposing Stage II to realistic artifacts; they are mixed with clean downsampled samples at a 7:3 ratio.

Main results

On VBench-I2V at (16,16,4)(16,16,4)0 with 81 frames, SwiftI2V achieves the best total score (6.4244) among compared methods, with the best I2V Background score (0.9975) and competitive I2V Subject fidelity (0.9910):

Model Total I2V Subject I2V Background Dynamic Degree
DiffVSR 6.4228 0.9878 0.9907 0.2917
Stream-DiffVSR 6.4240 0.9879 0.9933 0.3374
LTX-2 6.3579 0.9914 0.9932 0.0488
CineScale 6.3638 0.9924 0.9973 0.1667
SwiftI2V 6.4244 0.9910 0.9975 0.3008

Two caveats qualify these numbers. CineScale was evaluated on a randomly sampled 63-sample subset due to its cost (~50 minutes per sample even on 8 H800 GPUs); on that subset SwiftI2V leads on total score, and a Wilcoxon signed-rank test shows a statistically significant background-consistency advantage ((16,16,4)(16,16,4)1) while subject fidelity is statistically indistinguishable ((16,16,4)(16,16,4)2). The Dynamic Degree comparison also requires interpretation: Stream-DiffVSR's higher score reflects flickering rather than coherent motion, whereas LTX-2 and CineScale produce largely static outputs under strong image conditioning. Notably, Stage II raises Dynamic Degree from 0.2805 (Stage I output upsampled) to 0.3008, indicating that detail-preserving refinement does not suppress motion magnitude.

Efficiency

The headline efficiency result is a 202× reduction in total GPU-time relative to CineScale: 111 s on one H800 versus 5600 s across four GPUs (22,400 GPU·s). SwiftI2V is also faster than the efficiency-oriented LTX-2 on a single GPU (111 s vs. 152 s, a 1.37× speedup) with lower peak memory (33.5 GB vs. 45.7 GB). Stage II alone peaks at only 20.8 GB. With CPU offloading for Stage I, the full pipeline runs on a single RTX 4090 (24 GB), completing a sample in about 380 s — though flash attention could not be enabled in that environment, so this figure likely overstates the consumer-GPU latency.

CSG additionally enables streaming output. With DiT and VAE placed on two GPUs and decoding overlapped at segment boundaries, the first viewable 4-frame block appears after 8.5 s, with incremental delivery at roughly 3.33 fps. The paper concedes this throughput is below real-time playback and currently requires two GPUs.

Ablations

All ablations share Stage I and report Stage II metrics. Removing CSG (full-temporal denoising) increases peak memory from 20.8 GB to 22.1 GB and DiT time from 25 s to 30 s, with slightly lower quality. Replacing bidirectional interaction with causal AR masking (with KV caching) drops the total score from 6.424 to 6.392 despite faster DiT time (16 s vs. 25 s), because KV caching inflates peak memory and unidirectional flow amplifies cross-segment discontinuities. A segment-wise degradation analysis quantifies this: the AR variant degrades NIQE at +0.093/step (2.26% per step) versus +0.050/step (1.22%) for SwiftI2V, and its MUSIQ slope turns negative (−0.174) while SwiftI2V remains positive (+0.243). Removing stage-transition training similarly degrades results and leaves Stage I artifacts visible after refinement. Boundary-smoothness measurements show SwiftI2V's boundary-vs-non-boundary dissimilarity gaps are consistently smaller than those of a segmentation-free reference (e.g., +1.1% vs. +6.4% pixel difference), indicating no temporal popping at segment boundaries. Hyperparameter sweeps favor (16,16,4)(16,16,4)3; enlarging the conditioning window to (16,16,4)(16,16,4)4 dilutes the image anchoring and costs latency without quality gains.

Scalability experiments show that with CSG, Stage II peak memory stays below 24 GB even at 241 frames, with near-linear runtime growth in sequence length, whereas removing CSG yields sharply superlinear growth in both.

Limitations and open questions

The paper states several limitations plainly. SwiftI2V does not achieve real-time 2K synthesis; reaching it would require further advances in model compression, caching, scheduling, and hardware-aware inference. The two-stage modular design complicates production deployment relative to a single monolithic model, particularly regarding memory sharing, batching, and model loading. The CineScale comparison rests on a 63-sample subset, and the Dynamic Degree improvement over CineScale does not reach statistical significance ((16,16,4)(16,16,4)5, though effect size (16,16,4)(16,16,4)6). The streaming demonstration assumes the Stage I reference is already available and requires a second GPU for overlapped decoding. Open questions include whether unified distillation or shared components can simplify deployment, and whether the demonstrated streaming baseline can be pushed toward real-time rates through faster decoding or dedicated accelerators.

Conclusion

SwiftI2V demonstrates that a carefully instantiated two-stage decomposition — low-resolution motion generation followed by image-anchored, segment-wise 2K detail synthesis — resolves the efficiency–fidelity dilemma in high-resolution I2V more effectively than either end-to-end generation or generic VSR cascades. Its combination of bounded per-step token budgets, bidirectional within-window interaction, and stage-transition training yields competitive VBench-I2V quality at 2K with a 202× GPU-time reduction and single-consumer-GPU deployability, establishing a concrete foundation for scalable long-duration and streaming high-resolution I2V systems.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.