- The paper introduces SwiftI2V, a two-stage image-to-video system that combines low-resolution motion generation with image-anchored 2K detail synthesis and conditional segment-wise generation.
- SwiftI2V achieves a leading VBench-I2V score of 6.4244 at 2560×1408 resolution, including 0.9975 background fidelity and 0.9910 subject fidelity across 81-frame videos.
- The system reduces GPU time by 202× versus CineScale, runs in about 111 seconds on one H800 or approximately 380 seconds on an RTX 4090 with CPU offloading, and supports segmented streaming output.
Motivation and problem setting
High-resolution image-to-video (I2V) generation requires synthesizing plausible temporal dynamics while preserving the spatial structure and fine-grained textures of a given input image. At 2K resolution, this task is dominated by two coupled difficulties. First, token counts grow rapidly with resolution, making attention-based Diffusion Transformer (DiT) inference expensive in both memory and latency. Second, I2V imposes stricter fidelity requirements than text-to-video (T2V): appearance drift from the input image becomes less tolerable as resolution increases, since high-frequency details such as textures and identity cues must remain stable across frames.
The paper identifies two existing paradigms and their respective failures. End-to-end high-resolution generation with a single model couples global motion modeling and fine-detail synthesis in one objective, which demands large backbones, many sampling steps, and prohibitive GPU resources. The alternative low-resolution generation plus video super-resolution (LR+VSR) pipeline reduces cost but typically applies a generic VSR model that is not explicitly conditioned on the input image, leading to hallucinated details and drift from input-specific local structures. SwiftI2V is designed to occupy the favorable corner of this trade-off: end-to-end-level fidelity at LR+VSR-level cost.
Method
SwiftI2V follows the widely adopted motion–detail decoupled two-stage design, but its contribution lies in how each stage is instantiated for strongly image-conditioned 2K synthesis.
Stage I: low-resolution motion reference. The input image is downsampled to 360P (640×352), and a large-capacity DiT backbone (Wan2.1-I2V-480P) generates a low-resolution video serving as a motion and structure reference. A rank-128 LoRA adapts the backbone to 360P I2V, and an off-the-shelf few-step distillation LoRA enables 4-step sampling without classifier-free guidance.
Hybrid reference construction. Stage I's output is upsampled to target resolution, and its first frame is replaced by the full-resolution input image. This first-frame replacement injects the input as a boundary condition, reducing first-frame mismatch relative to conventional VSR pipelines while retaining Stage I's motion cues.
Stage II: conditional high-resolution synthesis. Rather than treating refinement as generic super-resolution, Stage II uses a smaller DiT (Wan2.2-TI2V-5B, fully fine-tuned) with a higher-compression 3D VAE (downsampling factors (16,16,4)). The hybrid reference latent zref is channel-concatenated with the noisy latent, whose first temporal block is replaced by the encoded input image zx, acting as an explicit appearance anchor. The patch-embedding input channels are expanded from 48 to 96, initialized by copying pretrained weights and zero-initializing the new channels. An appendix validates that this VAE preserves high-frequency content at 2K (PSNR 35.25 dB, SSIM 0.941), with power spectral density curves overlapping the input even above normalized frequency 0.6 — supporting the claim that aggressive compression does not sacrifice detail recovery capacity.
Conditional segment-wise generation
The central scalability mechanism is Conditional Segment-wise Generation (CSG). The DiT input is split along time into blocks; the first block is the image anchor, and the remaining t−1 blocks are partitioned into S=⌈(t−1)/M⌉ segments of M noisy blocks each. Each diffusion step processes only a window consisting of the anchor block, the last N finalized neighbor blocks, and the current segment, bounding the per-step 2K token budget. With (M,N)=(3,1), the 20 generative blocks of an 81-frame video are covered by seven segments.
Two design choices distinguish CSG from prior streaming approaches such as HiStream, CaSuVid, and MAGI-1:
- Bidirectional contextual interaction. Within each window, standard attention lets the anchor, neighbor, and noisy blocks attend to each other bidirectionally, rather than imposing a causal mask where history is read-only context. Crucially, updates are applied only to the current segment; conditioning latents are never written back, so finalized segments remain immutable. This avoids autoregressive error accumulation while still allowing context features to adapt dynamically during denoising.
- Teacher-forced training under CSG. Conditioning blocks are drawn from ground-truth videos during training, with loss computed only on noisy blocks, so the model learns to exploit anchored appearance and short-range context.
A third component addresses the cascade interface gap. Because Stage II is trained on clean downsampled clips but consumes Stage I outputs containing VAE distortion and flicker at inference, the authors synthesize Stage-I-like training inputs by adding noise (σ=0.10, one denoising step) to downsampled clips and denoising them with Stage I. These pairs preserve ground-truth motion correspondence while exposing Stage II to realistic artifacts; they are mixed with clean downsampled samples at a 7:3 ratio.
Main results
On VBench-I2V at (16,16,4)0 with 81 frames, SwiftI2V achieves the best total score (6.4244) among compared methods, with the best I2V Background score (0.9975) and competitive I2V Subject fidelity (0.9910):
| Model |
Total |
I2V Subject |
I2V Background |
Dynamic Degree |
| DiffVSR |
6.4228 |
0.9878 |
0.9907 |
0.2917 |
| Stream-DiffVSR |
6.4240 |
0.9879 |
0.9933 |
0.3374 |
| LTX-2 |
6.3579 |
0.9914 |
0.9932 |
0.0488 |
| CineScale |
6.3638 |
0.9924 |
0.9973 |
0.1667 |
| SwiftI2V |
6.4244 |
0.9910 |
0.9975 |
0.3008 |
Two caveats qualify these numbers. CineScale was evaluated on a randomly sampled 63-sample subset due to its cost (~50 minutes per sample even on 8 H800 GPUs); on that subset SwiftI2V leads on total score, and a Wilcoxon signed-rank test shows a statistically significant background-consistency advantage ((16,16,4)1) while subject fidelity is statistically indistinguishable ((16,16,4)2). The Dynamic Degree comparison also requires interpretation: Stream-DiffVSR's higher score reflects flickering rather than coherent motion, whereas LTX-2 and CineScale produce largely static outputs under strong image conditioning. Notably, Stage II raises Dynamic Degree from 0.2805 (Stage I output upsampled) to 0.3008, indicating that detail-preserving refinement does not suppress motion magnitude.
Efficiency
The headline efficiency result is a 202× reduction in total GPU-time relative to CineScale: 111 s on one H800 versus 5600 s across four GPUs (22,400 GPU·s). SwiftI2V is also faster than the efficiency-oriented LTX-2 on a single GPU (111 s vs. 152 s, a 1.37× speedup) with lower peak memory (33.5 GB vs. 45.7 GB). Stage II alone peaks at only 20.8 GB. With CPU offloading for Stage I, the full pipeline runs on a single RTX 4090 (24 GB), completing a sample in about 380 s — though flash attention could not be enabled in that environment, so this figure likely overstates the consumer-GPU latency.
CSG additionally enables streaming output. With DiT and VAE placed on two GPUs and decoding overlapped at segment boundaries, the first viewable 4-frame block appears after 8.5 s, with incremental delivery at roughly 3.33 fps. The paper concedes this throughput is below real-time playback and currently requires two GPUs.
Ablations
All ablations share Stage I and report Stage II metrics. Removing CSG (full-temporal denoising) increases peak memory from 20.8 GB to 22.1 GB and DiT time from 25 s to 30 s, with slightly lower quality. Replacing bidirectional interaction with causal AR masking (with KV caching) drops the total score from 6.424 to 6.392 despite faster DiT time (16 s vs. 25 s), because KV caching inflates peak memory and unidirectional flow amplifies cross-segment discontinuities. A segment-wise degradation analysis quantifies this: the AR variant degrades NIQE at +0.093/step (2.26% per step) versus +0.050/step (1.22%) for SwiftI2V, and its MUSIQ slope turns negative (−0.174) while SwiftI2V remains positive (+0.243). Removing stage-transition training similarly degrades results and leaves Stage I artifacts visible after refinement. Boundary-smoothness measurements show SwiftI2V's boundary-vs-non-boundary dissimilarity gaps are consistently smaller than those of a segmentation-free reference (e.g., +1.1% vs. +6.4% pixel difference), indicating no temporal popping at segment boundaries. Hyperparameter sweeps favor (16,16,4)3; enlarging the conditioning window to (16,16,4)4 dilutes the image anchoring and costs latency without quality gains.
Scalability experiments show that with CSG, Stage II peak memory stays below 24 GB even at 241 frames, with near-linear runtime growth in sequence length, whereas removing CSG yields sharply superlinear growth in both.
Limitations and open questions
The paper states several limitations plainly. SwiftI2V does not achieve real-time 2K synthesis; reaching it would require further advances in model compression, caching, scheduling, and hardware-aware inference. The two-stage modular design complicates production deployment relative to a single monolithic model, particularly regarding memory sharing, batching, and model loading. The CineScale comparison rests on a 63-sample subset, and the Dynamic Degree improvement over CineScale does not reach statistical significance ((16,16,4)5, though effect size (16,16,4)6). The streaming demonstration assumes the Stage I reference is already available and requires a second GPU for overlapped decoding. Open questions include whether unified distillation or shared components can simplify deployment, and whether the demonstrated streaming baseline can be pushed toward real-time rates through faster decoding or dedicated accelerators.
Conclusion
SwiftI2V demonstrates that a carefully instantiated two-stage decomposition — low-resolution motion generation followed by image-anchored, segment-wise 2K detail synthesis — resolves the efficiency–fidelity dilemma in high-resolution I2V more effectively than either end-to-end generation or generic VSR cascades. Its combination of bounded per-step token budgets, bidirectional within-window interaction, and stage-transition training yields competitive VBench-I2V quality at 2K with a 202× GPU-time reduction and single-consumer-GPU deployability, establishing a concrete foundation for scalable long-duration and streaming high-resolution I2V systems.