SwiftVR: Real-Time Generative Video Restoration

• SwiftVR is a streaming, one-step generative video restoration framework that uses a causal chunk-wise protocol to meet real-time streaming demands on both high-end and consumer GPUs.
• It leverages innovative Mask-free Shifted-window Self-Attention (MFSWA) and a Restoration-aware Autoencoder (ReAE) to overcome quadratic spatial attention costs and autoencoder memory overhead.
• The framework delivers up to 31 FPS at 2560×1440 and 26 FPS at 1080p, marking the first real-time 1080p generative video restoration on consumer-grade hardware.

Searching arXiv for SwiftVR and closely related video restoration papers. SwiftVR is a streaming, one-step generative video restoration framework for real-time restoration of live video streams at high resolution. It targets a deployment regime defined by strict per-frame latency, causal inference, and single-GPU memory constraints, and it addresses two bottlenecks identified for one-step diffusion-based video restoration: quadratic spatial attention at high resolutions and the latency-memory overhead of large video autoencoders. Under a causal chunk-wise protocol, SwiftVR sustains 31~FPS at 2560x1440 and 14~FPS at 3840x2160 on a single H100, and reaches 26~FPS at 1920x1080 on a consumer RTX~5090. The paper states that it is the first generative video restoration model to achieve real-time 1080p streaming on a consumer-grade GPU (Yan et al., 8 Jun 2026).

1. Nomenclature and scope

The term SwiftVR in the current literature refers explicitly to “SwiftVR: Real-Time One-Step Generative Video Restoration” (Yan et al., 8 Jun 2026). In that context, VR denotes video restoration, not virtual reality. This distinction is important because several adjacent papers use the acronym “VR” in the virtual-reality sense and are therefore topically related only by acronym or by broad systems concerns.

Paper	Topic	Relation to SwiftVR
(Yan et al., 8 Jun 2026)	Real-time one-step generative video restoration	Defines SwiftVR
(Li et al., 2024)	StereoID for stereoscopic visual inconsistencies in VR apps	No explicit SwiftVR mention
(Lin, 2020)	Virtual-to-real mirror/window interaction paradigm	Not named SwiftVR
(Royston et al., 2016)	GraphiteVR collaborative untethered VR environment	Same broad VR class, not SwiftVR

The 2024 paper on stereoscopic rendering quality introduces StereoID, an unsupervised black-box testing framework for detecting stereoscopic visual inconsistencies (SVI) in VR apps, and the accompanying summary states that there is no explicit mention of SwiftVR as a framework, tool, baseline, or related system (Li et al., 2024). The 2020 paper “Keep It Real: a Window to Real Reality in Virtual Reality” proposes a virtual-to-real (V2R) interaction paradigm based on a virtual mirror or window with correct perspective geometry, but it likewise does not define SwiftVR (Lin, 2020). The 2016 collaborative visualization paper describes GraphiteVR, a room-scale, untethered, multi-user VR environment; the supplied summary treats it only as being in the same general class of systems rather than as SwiftVR itself (Royston et al., 2016).

This suggests that SwiftVR should be understood narrowly as the 2026 video restoration framework unless a different disambiguation is supplied.

2. Problem setting and design objective

SwiftVR is formulated for real-world video restoration for live streams such as broadcasting, conferencing, and cloud gaming. The targeted setting is harder than offline restoration because the system must be causal, must sustain display-rate throughput, must fit within a single-GPU memory budget, and must handle unknown and time-varying degradations such as blur, noise, compression artifacts, and motion corruption (Yan et al., 8 Jun 2026).

The paper positions existing approaches along a familiar tradeoff. Traditional regression-based restoration methods are efficient but tend to produce overly smooth outputs. Multi-step diffusion methods improve perceptual quality but are too slow for streaming deployment. One-step diffusion is therefore treated as the correct direction for streaming, but the paper argues that two residual bottlenecks remain: quadratic self-attention cost at high resolution and autoencoder latency/memory overhead, especially from large video VAEs. Using the representative backbone Wan2.2-TI2V-5B, the paper reports attention compute increasing from 0.47 PFLOPs at 720p to 21.3 PFLOPs at 4K in the representative setup (Yan et al., 8 Jun 2026).

A central design principle is deployability on mainstream hardware. SwiftVR therefore avoids future-frame access, avoids iterative diffusion sampling, avoids custom sparse kernels, and keeps attention on the standard dense SDPA path. A plausible implication is that the framework’s contribution is as much about systems-compatible reformulation as about restoration quality itself.

3. Causal chunk-wise streaming pipeline

SwiftVR operates under a causal chunk-wise streaming protocol. The input stream is divided into non-overlapping temporal chunks of fixed length; each chunk is encoded by the streaming ReAE encoder, restored by a single DiT forward pass, decoded by the streaming ReAE decoder, and then emitted immediately, with continuity maintained through cached boundary states rather than through long-context temporal attention (Yan et al., 8 Jun 2026).

The paper gives the chunk-wise inference rule as

$$Z_k^{HQ} = Z_k^{LQ} - v_\theta(E_{str}(X_k), 1), \qquad \hat{X}_k = D_{str}(Z_k^{HQ}, s_{k-1}).$$

This formulation makes the one-step character explicit: the model encodes the low-quality chunk, predicts a velocity-like latent correction, subtracts it, and decodes the restored result using streaming state from the previous chunk. The chunking strategy is motivated by the attention scaling law $O(N^2)$ with $N = T H W$; by bounding temporal length $T$, the method limits temporal growth and shifts the optimization problem toward spatial efficiency (Yan et al., 8 Jun 2026).

The paper also emphasizes what the framework does not do. There is no iterative diffusion sampling, no access to future chunks, and no rolling KV cache used to reduce the main attention cost. Continuity is handled by the autoencoder’s streaming states. This architecture differentiates SwiftVR from offline generative restoration pipelines that depend on bidirectional context or multi-step denoising.

4. Core architectural mechanisms

SwiftVR integrates two named mechanisms: Mask-free shifted-window self-attention (MFSWA) and a Restoration-aware Autoencoder (ReAE) (Yan et al., 8 Jun 2026).

Mask-free shifted-window self-attention

MFSWA is the framework’s primary solution to the attention bottleneck. Standard windowed transformers often rely on attention masks, cyclic shifts, padding, variable-sized boundary windows, or special sparse kernels, but the paper argues that these choices can disable optimized dense scaled-dot-product attention backends on consumer GPUs. SwiftVR instead keeps every attention call on the standard dense SDPA interface with attn_mask = None, no padding tokens, no ragged tensors, and no cyclic shift inside the attention kernel. Window structure is implemented through deterministic gather/scatter index tensors before and after attention (Yan et al., 8 Jun 2026).

The design has three parts: Spatial-only partitioning, Dense-block pre-gathering, and Half-window shift with priority-coherent scatter. Attention is applied to dense tensors of approximate size

$T \times w_h \times w_w$

rather than to the full $T H W$ token grid. The paper argues that this preserves the dense path and thereby backend portability across PyTorch SDPA, FlashAttention-2, FlashAttention-3, xFormers, and SageAttention (Yan et al., 8 Jun 2026).

Restoration-aware Autoencoder

The second mechanism, ReAE, addresses the latency and memory cost of the video autoencoder. The paper characterizes the original Wan2.2 VAE as too expensive for real-time high-resolution streaming, while a tiny autoencoder is fast but causes a severe reconstruction-quality drop. ReAE is introduced as a compromise that is lightweight enough for streaming and still preserves reconstruction quality sufficiently for restoration (Yan et al., 8 Jun 2026).

The ablation summary is explicit. Wan2.2-VAE has 704.69M parameters, 24.86 GB peak memory, and 2.714 s decoding time. The tiny autoencoder has 11.42M params and 0.040 s decoding but much weaker PSNR/LPIPS. ReAE has 40.95M params, 16.97 GB peak memory, 0.034 s encoding, and 0.099 s decoding, with substantially better quality than the tiny autoencoder (Yan et al., 8 Jun 2026).

Taken together, MFSWA and ReAE define the paper’s systems thesis: local attention must remain compatible with dense SDPA backends, and the latent codec must be lightweight enough that encode/decode no longer dominates one-step inference.

5. Training formulation

SwiftVR is trained in three stages: latent flow matching with full attention, MFSWA distillation, and joint pixel-space fine-tuning (Yan et al., 8 Jun 2026).

In the first stage, high-quality and low-quality videos are encoded as

$Z^{HQ} = E_d(X^{HQ}), \qquad Z^{LQ} = E_d(X^{LQ}),$

and interpolated along

$z_t = (1-t)Z^{HQ} + tZ^{LQ}, \qquad t \in [0,1].$

The target is the constant velocity $Z^{LQ} - Z^{HQ}$, trained with the flow-matching loss

$$\mathcal{L}_{FM} = \mathbb{E}\left\| v_\theta(z_t, t) - (Z^{LQ} - Z^{HQ}) \right\|_2^2.$$

In the second stage, the full-attention teacher is distilled into the shifted-window student using

$O(N^2)$0

In the third stage, the DiT and ReAE are jointly fine-tuned under the deployment-time inference protocol. The paper writes the latent update and decode step as

$O(N^2)$1

with the supervision objective

$O(N^2)$2

ReAE itself is trained independently on clean video in two stages. The reconstruction stage uses

$O(N^2)$3

where $O(N^2)$4 is pixel $O(N^2)$5 loss, $O(N^2)$6 is perceptual similarity loss, and $O(N^2)$7 is a temporal consistency term based on differences between consecutive frame differences. Adversarial fine-tuning then adds

$O(N^2)$8

The training program reflects the paper’s separation of concerns: first learn a direct one-step latent correction, then transfer it into the efficient attention geometry, and finally align latent behavior with pixel-space deployment.

6. Benchmarks, efficiency, and limitations

SwiftVR is evaluated on SPMCS, UDM10, YouHQ40, and VideoLQ, with comparisons against Real-ESRGAN, RealBasicVSR, RealViFormer, Upscale-A-Video, DOVE, SeedVR2-3B, and FlashVSR-Tiny (Yan et al., 8 Jun 2026). The paper reports that SwiftVR ranks first in MUSIQ on all four benchmarks, ranks first in CLIP-IQA and MANIQA on UDM10 and YouHQ40, is competitive in LPIPS/DISTS, and restores details such as feather texture, foliage structure, fence lines, branch boundaries, and car contours more naturally than many baselines (Yan et al., 8 Jun 2026).

The reported throughput numbers are central to the paper’s significance.

Hardware / setting	Resolution	Throughput
Single H100-80G, causal streaming	2560 × 1440	31.32 FPS
Single H100-80G, causal streaming	3840 × 2160	13.84 FPS
Consumer RTX 5090, causal streaming	1920 × 1080	26 FPS

At 2560×1440, the paper reports SwiftVR: 0.766 s per 24-frame chunk, 31.32 FPS, 38.01 GB; FlashVSR-Tiny: 2.493 s, 9.61 FPS, 34.35 GB; SeedVR2-3B: 17.320 s, 1.39 FPS, 35.35 GB; and DOVE: 27.615 s, 0.87 FPS, 59.24 GB. It states that SwiftVR is about 3.3× faster than FlashVSR-Tiny and that all compared diffusion-based baselines exceed the memory limit at 4K, whereas SwiftVR reports 60.91 GB peak memory at 4K on H100 (Yan et al., 8 Jun 2026).

The ablations are aligned with the framework’s main claims. For attention, full attention reaches 19.36 FPS, masked shifted-window attention reaches 27.47 FPS, and MFSWA reaches 31.32 FPS. The masked version is slower because the attention mask disables fused SDPA and forces a slower execution path. The supplementary material also notes that boundary-clamped gather overhead increases compute somewhat near boundaries, but does not significantly inflate peak memory, decreases at higher resolutions, and remains worthwhile because the windows stay dense and SDPA-compatible (Yan et al., 8 Jun 2026).

The main limitation identified by the authors is that SwiftVR is not yet real-time 4K on consumer GPUs. The paper lists inference-time acceleration, post-training quantization, token reduction, KV-state compression/caching, and smaller, more compact backbones as future directions (Yan et al., 8 Jun 2026). This suggests that SwiftVR should be regarded as a deployment-oriented milestone in generative video restoration rather than as an endpoint: it establishes real-time 1080p consumer deployment and 4K single-H100 feasibility, while leaving consumer-grade 4K streaming as an open systems problem.