SwiftVR: Real-Time One-Step Generative Video Restoration

Abstract: Real-time video restoration (VR) for live streams requires high-resolution outputs under strict per-frame latency constraints. Existing one-step diffusion-based VR models remain difficult to deploy on consumer-grade GPUs due to two main bottlenecks: quadratic spatial attention at high resolutions and the latency-memory overhead of large video autoencoders. We present SwiftVR, a streaming one-step generative VR framework that reduces both bottlenecks under a causal chunk-wise protocol. For attention, mask-free shifted-window self-attention gathers each spatial window into a dense tensor via deterministic indexing, keeping all attention calls on the dense scaled dot-product attention path without masks, cyclic shifts, padding, or hardware-specific sparse kernels. Because SwiftVR uses only standard dense SDPA calls, the trained model transfers to consumer GPUs without retraining or custom kernels. For autoencoding, a lightweight Restoration-aware Autoencoder enables fast chunk-wise decoding while preserving reconstruction quality. On a single H100, SwiftVR sustains 31~FPS at 2560x1440 and 14~FPS at 3840x2160, whereas all compared diffusion-based VR baselines exceed the memory limit at 4K. On a consumer RTX~5090, SwiftVR reaches 26~FPS at 1920x1080. To our knowledge, SwiftVR is the first generative VR model to achieve real-time 1080p streaming on a consumer-grade GPU, while attaining strong no-reference perceptual quality with lower inference cost. Project is available at https://h-oliday.github.io/SwiftVR.

• The paper presents SwiftVR, a one-step generative video restoration method that leverages mask-free shifted-window self-attention to optimize throughput.
• It employs a restoration-aware autoencoder jointly trained with DiT, reducing inference latency and memory usage without compromising perceptual quality.
• SwiftVR achieves real-time 1080p+ streaming on consumer GPUs and competitive perceptual metrics, paving the way for scalable live video restoration.

SwiftVR: Real-Time One-Step Generative Video Restoration

Problem Formulation and Motivation

Generative video restoration (VR) for live streams at display resolution demands strict frame-rate and latency constraints, particularly for real-world deployment on consumer-grade hardware. Recent methods have demonstrated strong perceptual quality using diffusion-based and generative approaches, mostly via multi-step denoising or complex attention modules, which introduce prohibitive memory and computational overheads—making real-time application at high resolution impractical. One-step diffusion models reduce generative sampling cost to a single network evaluation, but attention scaling and autoencoder bottlenecks persist. Quadratic spatial attention and heavyweight autoencoders prevent deployment at 1080p+ on resource-constrained hardware. Addressing these dual bottlenecks is critical for practical, scalable, and perceptually strong video restoration.

Methodology

SwiftVR introduces a unified architecture and protocol for real-time, streaming, one-step generative video restoration, with robust cross-backend, cross-hardware deployment. Key innovations are:

1. Mask-Free Shifted-Window Self-Attention (MFSWA):

Spatial attention employs deterministic window partitioning and half-window spatial shifts via index-based gather/scatter. Windows are processed as dense tensors through standard SDPA, maintaining optimal throughput on hardware-accelerated fused attention paths (e.g., cuDNN/FlashAttention/xFormers). This window-local, mask-free design removes the necessity for cyclic shifts, dynamic masking, or padding tokens, all of which force fallback to slower paths and complicate cross-backend deployment. MFSWA ensures locality while fully leveraging efficient dense attention backends. The attention operation is confined to spatial axes, as chunked-causal streaming bounds temporal growth. Measured, MFSWA achieves 1.62$\times$ the throughput of an equivalent full-attention backbone with negligible perceptual degradation.

2. Restoration-aware Autoencoder (ReAE):

The autoencoder interface employs a lightweight, fine-tuned latent model jointly optimized with the DiT backbone. The ReAE achieves favorable trade-offs: it dramatically reduces parameter count and inference latency relative to high-fidelity, high-cost backbones (e.g., Wan2.2-VAE), without the quality collapse of minimal architectures. Joint fine-tuning with the DiT in pixel space closes the latent-pixel gap associated with previous dual-stage training.

3. Causal Chunk-wise Streaming Protocol:

Streaming inference is decomposed into fixed-length, non-overlapping chunks with causal encoder/decoder state propagation, strictly disallowing access to future frames. All baselines are rebenchmarked under a unified, causal streaming protocol to ensure fair comparison.

Optimization proceeds in three stages: full-attention DiT training for latent flow matching, window-distilled student distillation, and joint adversarial fine-tuning with the ReAE and a multi-scale VGG-based video discriminator.

Quantitative and Qualitative Results

Throughput and Efficiency:

SwiftVR delivers unprecedented performance among generative VR methods:

• 4K UHD (3840×2160): 14 FPS on a single H100-80GB (all baseline generative models are out-of-memory at this resolution).
• 2560×1440 (QHD): 31 FPS on H100; 1920×1080 (FHD): 54 FPS.
• Consumer RTX 5090, 1080p: 26 FPS, the first reported real-time generative VR deployment at this resolution on consumer hardware.
• By comparison, FlashVSR-Tiny achieves 9.6 FPS at 2560×1440, DOVE and SeedVR2-3B are nearly an order of magnitude slower.

Memory Efficiency:

SwiftVR maintains manageable peak memory (e.g., 38 GB at 2560×1440 on H100), in contrast to baseline DiT- and VAE-based models which require >59 GB and tile-based partitioning, forfeiting practical throughput.

Fidelity and Perceptual Quality:

Across synthetic and real-world benchmarks (SPMCS, UDM10, YouHQ40, VideoLQ):

• SwiftVR consistently achieves SOTA or near-SOTA on no-reference metrics (MUSIQ, CLIP-IQA, MANIQA) and is competitive (<0.01 difference) on LPIPS/DISTS.
• While pixel-oriented fidelity metrics (PSNR/SSIM) sometimes favor regression baselines, perceptual metrics show superiority or parity with both one-step and multi-step diffusion competitors.
• Qualitatively, SwiftVR reconstructions preserve fine textures, color fidelity, and structural details—outperforming smoothing-biased regression baselines and artifact-prone generative baselines.

Ablation Studies:

Removal of MFSWA or replacement with mask-based windowing disables the high-throughput dense path; encoding window structure externally is essential for practical speedup. Employing the lightweight ReAE reduces decoding time and peak memory without eroding perceptual quality.

Deployment and Generality

Cross-backend Support:

Due to strict adherence to dense SDPA, SwiftVR deploys unmodified (with matching fidelity/throughput) on standard PyTorch SDPA, cuDNN, FlashAttention-2/3, SageAttention, and xFormers. No retraining or kernel rewriting is required, and the same weights transfer seamlessly from H100 to consumer RTX.

Scalability Limits:

Real-time 4K UHD restoration on consumer GPUs remains infeasible with present resource trade-offs. Significant headroom for acceleration exists through further compression, quantization, or architecture pruning.

Implications and Future Directions

SwiftVR sets an architectural and systems paradigm for practical, perceptually optimized, real-time generative video restoration on commodity hardware. The demonstration that mask-free, windowed self-attention enables efficient, hardware-agnostic inference at 1080p+ resolutions points toward future, more compact DiT/Autoencoder backbones. SwiftVR’s efficient streaming protocol is compatible with rolling-state and windowed context acceleration, paving the way for longer-horizon video contexts and even higher resolutions.

Future developments in this domain may focus on:

• Inference-side acceleration: post-training quantization, learned token pruning, or more aggressive KV caching.
• Base model compression: employing lighter generative backbones while preserving generative fidelity for even higher resolution or lower-end deployment.
• Semi-supervised/unsupervised fine-tuning: adapting to unknown, real-world degradations and domain-shifted input statistics.

Conclusion

SwiftVR operationalizes generative video restoration for real-world, live-streaming deployment, overcoming the dual computational obstacles of spatial attention scaling and decoder latency. Mask-free shifted-window attention and a restoration-aware autoencoder establish the practical, scalable foundation for streaming video restoration with high perceptual fidelity on both server-class and consumer GPUs. While true real-time 4K restoration on consumer hardware awaits further research advances, SwiftVR provides a flexible, high-quality, and efficient backbone for future generative video enhancement systems (2606.09516).