RealisVSR: Real-World 4K Video Super-Resolution

• The paper introduces RealisVSR, a diffusion-based method leveraging Wan2.1 to achieve improved PSNR, SSIM, and temporal consistency over prior VSR approaches.
• It incorporates a Consistency Preserved ControlNet and a High-Frequency Rectified Diffusion Loss to restore fine textures and suppress artifact propagation.
• The work establishes new benchmarks, including RealisVideo-4K, to rigorously evaluate ultra-high-resolution detail recovery in real-world video scenarios.

Searching arXiv for the RealisVSR paper and closely related VSR context. RealisVSR is a real-world 4K video super-resolution method that formulates VSR as a high-frequency detail-enhanced video diffusion problem built on Wan2.1. It is introduced to address three stated limitations in the contemporary VSR literature: inconsistent modeling of temporal dynamics in foundational models, limited high-frequency detail recovery under complex real-world degradations, and insufficient evaluation of detail enhancement and 4K super-resolution because current methods primarily rely on 720P datasets with inadequate details (Zhao et al., 25 Jul 2025). The method combines a Consistency Preserved ControlNet, a High-Frequency Rectified Diffusion Loss, and a new benchmark, RealisVideo-4K, and is evaluated on REDS, SPMCS, UDM10, YouTube-HQ, VideoLQ, and RealisVideo-720P, with particular emphasis on ultra-high-resolution scenarios (Zhao et al., 25 Jul 2025).

1. Problem setting and motivation

Video super-resolution seeks to reconstruct high-resolution video from low-resolution input while preserving both per-frame fidelity and temporal coherence. In the real-world setting emphasized by RealisVSR, the degradation process is mixed and includes compression, blur, and sensor artifacts rather than a single synthetic corruption model (Zhao et al., 25 Jul 2025). The paper argues that this setting exposes weaknesses in existing diffusion-based VSR systems: although such methods can generate plausible textures, they remain limited by weak temporal modeling and insufficient recovery of fine high-frequency detail (Zhao et al., 25 Jul 2025).

A central claim of the work is that the evaluation ecosystem itself has constrained progress. Existing datasets such as REDS, SPMCS, UDM10, YouTube-HQ, and Vid4 are described as mostly 720P, and their ground-truth videos are said to lack the fine detail needed to evaluate hair, skin texture, fabric, or small objects at the level demanded by 4K super-resolution (Zhao et al., 25 Jul 2025). This motivates not only a new model, but also a new benchmark designed to expose failures that remain hidden at lower resolution.

The paper situates RealisVSR within the broader transition from GAN-based VSR to diffusion-based VSR. It states that diffusion models have effectively addressed the over-smoothing issues inherent in GAN-based methods, but that recent advances still leave open the problems of temporal consistency and fine-detail restoration under real degradations (Zhao et al., 25 Jul 2025). This suggests that the method is intended not as a generic diffusion baseline, but as a targeted adaptation of a video diffusion prior to the specific demands of super-resolution.

2. Architectural foundation in Wan2.1

RealisVSR is built on Wan2.1, which the authors treat as a stronger spatio-temporal video diffusion prior than earlier backbones such as CogVideoX or I2VGen-XL (Zhao et al., 25 Jul 2025). The standard Wan2.1 pipeline serves as the “fundamental model,” and is described as consisting of a 3D causal VAE, DiT blocks, and the umT5 text encoder (Zhao et al., 25 Jul 2025).

The input video is denoted

$$\mathcal{V} \in \mathbb{R}^{(1+T)\times H\times W\times 3},$$

and is encoded by Wan-VAE into compressed latent dimensions $[1+T/4, H/8, W/8]$, with channel expansion from $C=3$ to $C'=16$ (Zhao et al., 25 Jul 2025). The first frame receives only spatial compression, which the paper states preserves fidelity for image-conditioned restoration (Zhao et al., 25 Jul 2025). Within the DiT backbone, patchifying is implemented by 3D convolution to form spatio-temporal tokens, followed by transformer blocks with cross-attention for text conditioning and an unpatchifying stage (Zhao et al., 25 Jul 2025).

This backbone choice is significant because RealisVSR’s claims about temporal consistency depend on the underlying video prior rather than on a framewise restoration strategy. The paper explicitly attributes part of its temporal gains to Wan2.1, arguing that it provides superior temporal modeling relative to earlier foundational models (Zhao et al., 25 Jul 2025). A plausible implication is that the method’s effectiveness depends not only on the added losses and conditioning mechanisms, but also on inheriting motion-aware priors from a pretrained video generator.

3. Consistency Preserved ControlNet

The first of the paper’s three core innovations is the Consistency Preserved ControlNet, or CPC, which injects low-quality video information into the denoising process (Zhao et al., 25 Jul 2025). In standard ControlNet, both the noisy latent $$z_t \in \mathbb{R}^{B\times C\times T\times H\times W}$$ and the condition signals are combined in the conditioning path. RealisVSR modifies this design by removing the noisy latent from the ControlNet branch entirely (Zhao et al., 25 Jul 2025).

The contrast is expressed as

$$x_{\text{comb}} = \mathcal{T}_{\theta_{\text{patch}}}(z_t) + \mathcal{T}_{\theta_{\text{cond}}}(c_t),$$

for vanilla ControlNet, whereas CPC uses only the conditional input:

$$x_{\text{CPC}} = \mathcal{T}_{\theta_{\text{cond}}}(c_t).$$

The authors report that this improves visual quality because it suppresses artifact propagation from the noisy latent while preserving convergence (Zhao et al., 25 Jul 2025).

CPC features are fused into the main denoising network through depth-adaptive residual connections. If the main network has $L_{\text{main}}$ blocks and CPC has $L_{\text{CPC}}$, then

$$r = \lceil L_{\text{main}} / L_{\text{CPC}} \rceil,$$

and the CPC feature is injected at main block $[1+T/4, H/8, W/8]$0 through

$[1+T/4, H/8, W/8]$1

where $[1+T/4, H/8, W/8]$2 is a learnable scaling factor (Zhao et al., 25 Jul 2025). The paper interprets this as multi-scale conditioning aligned across network depths.

Empirically, the ablation isolates CPC as primarily a consistency- and structure-oriented component. Starting from a baseline of 26.54 PSNR / 0.7843 SSIM / 0.1807 LPIPS / 0.0875 DISTS, adding CPC alone yields 26.58 / 0.8099 / 0.1520 / 0.0839 (Zhao et al., 25 Jul 2025). The quantitative pattern is described as showing that CPC mainly improves consistency and structure even when PSNR changes only slightly (Zhao et al., 25 Jul 2025).

4. High-Frequency Rectified Diffusion Loss

The second major contribution is the High-Frequency Rectified Loss, or HR-Loss, which augments the rectified-flow-style diffusion objective with explicit high-frequency constraints (Zhao et al., 25 Jul 2025). The paper begins from the diffusion formulation

$[1+T/4, H/8, W/8]$3

where $[1+T/4, H/8, W/8]$4 is the clean video and $[1+T/4, H/8, W/8]$5 is standard Gaussian noise (Zhao et al., 25 Jul 2025).

The underlying rectified-flow-style loss is written as

$[1+T/4, H/8, W/8]$6

and also more explicitly in the narrative as

$[1+T/4, H/8, W/8]$7

The paper states that this preserves structural fidelity, but is biased toward low-frequency structural accuracy and does not explicitly encourage high-frequency restoration (Zhao et al., 25 Jul 2025).

HR-Loss therefore adds two auxiliary terms. The first is a wavelet-based frequency decomposition loss:

$[1+T/4, H/8, W/8]$8

with $[1+T/4, H/8, W/8]$9, $C=3$0 applying Haar wavelet kernels, and

$C=3$1

The sub-band weights are set to

$C=3$2

so that the high-frequency bands LH, HL, and HH receive greater emphasis than the low-frequency LL band (Zhao et al., 25 Jul 2025).

The second term is a HOG-based texture constraint:

$C=3$3

The paper specifies 9 orientation bins across $C=3$4, with each bin covering $C=3$5, and magnitudes normalized via L2-Hys regularization (Zhao et al., 25 Jul 2025). This term is intended to encourage recovery of directional textures such as hair strands, edges, and fabric weave.

The full objective is

$C=3$6

The ablation attributes distinct roles to the two high-frequency terms. CPC plus wavelet loss reaches 27.24 / 0.8114 / 0.1549 / 0.0817, CPC plus HOG reaches 27.33 / 0.8141 / 0.1521 / 0.0787, and the full model reaches 27.36 / 0.8169 / 0.1388 / 0.0780 (Zhao et al., 25 Jul 2025). The paper interprets this as evidence that HOG is especially strong for structure-specific high-frequency cues, while wavelets provide more direct multiscale frequency correction (Zhao et al., 25 Jul 2025). A wavelet-weight sweep further reports 26.08 / 0.8099 / 0.1520 / 0.0839 at weight 1.0, 26.67 / 0.8000 / 0.1611 / 0.0875 at 1.5, 27.24 / 0.8114 / 0.1549 / 0.0817 at 2.0, and 26.19 / 0.7476 / 0.1834 / 0.0999 at 3.0, indicating that moderate emphasis on high-frequency bands is optimal within the reported setting (Zhao et al., 25 Jul 2025).

5. RealisVideo-4K and the evaluation regime

The third core contribution is RealisVideo-4K, described as the first public 4K VSR benchmark containing 1,000 high-definition video-text pairs (Zhao et al., 25 Jul 2025). The paper argues that previous VSR benchmarks are fundamentally limited by resolution, and therefore insufficient for evaluating true detail reconstruction at 4K (Zhao et al., 25 Jul 2025).

The dataset is collected and filtered from about 1,000 4K videos with rich detail, with 140 videos randomly selected as the test set (Zhao et al., 25 Jul 2025). Its collection process is two-stage: QWen2.5-VL is used to annotate videos with text, then keywords such as “close-up” and “detailed” are used to find candidate clips, followed by a manual filtering step that removes wrongly selected low-quality videos (Zhao et al., 25 Jul 2025). The benchmark is paired with RealisVideo-720P, obtained by resizing the 4K videos to 720P so that the lower-resolution benchmark still retains detail originating from 4K source material (Zhao et al., 25 Jul 2025).

This design is central to the paper’s empirical claims. RealisVideo-720P is described as a harder and more realistic test than standard 720P benchmarks because the super-resolution system must reconstruct details that truly existed in the original 4K source (Zhao et al., 25 Jul 2025). This suggests that the benchmark is intended to decouple apparent success on conventional 720P datasets from actual recovery of fine detail.

The experimental suite combines synthetic 720P benchmarks, a real-world no-reference benchmark, and the 4K benchmark. Full-reference evaluation is reported on SPMCS, UDM10, REDS30, YouTube-HQ, and RealisVideo-720P; no-reference metrics are used on VideoLQ; and the strongest evidence for ultra-high-resolution capability is presented on RealisVideo-4K (Zhao et al., 25 Jul 2025).

6. Training protocol, efficiency, and empirical results

Training is explicitly presented as data-efficient. RealisVSR is trained on a self-collected 4K dataset containing only 50K text-video pairs, which the paper states is just 5–25% of the training data used by prior diffusion-based methods (Zhao et al., 25 Jul 2025). In the main setup, videos are randomly cropped to a 480P-scale resolution for training and inference efficiency using a predefined 480P resolution bucket with varying aspect ratios, and LR-HR pairs are constructed with a two-order video degradation pipeline following mainstream real-world VSR practice (Zhao et al., 25 Jul 2025).

The implementation uses Wan-1.3B as the base model, trains CPC on top of it, and uses FSDP, a sequence degree of 2, 8 NVIDIA H20-100G GPUs, 12K iterations, batch size 8 with gradient accumulation, AdamW, weight decay $C=3$7, and learning rate $C=3$8 (Zhao et al., 25 Jul 2025). The supplement also reports inference efficiency: for 49-frame 720P output, Ours-480P takes 216s, compared with 510s for UAV, 1280s for MGLD-VSR, and 321s for STAR; a 720P-trained variant runs in 86s, faster than SeedVR’s 108s, while using about 1.6B parameters versus SeedVR’s 3B (Zhao et al., 25 Jul 2025).

The reported quantitative performance is summarized below.

Benchmark	Reported results
SPMCS	27.36 PSNR, 0.8169 SSIM, 0.1388 LPIPS, 0.0780 DISTS, 51.68 DOVER
UDM10	29.23 PSNR, 0.8708 SSIM, 0.1221 LPIPS, 0.0683 DISTS, 58.93 DOVER
REDS30	26.56 PSNR, 0.7593 SSIM, 0.2500 LPIPS, 0.0991 DISTS, 44.86 DOVER
YouTube-HQ	25.84 PSNR, 0.7601 SSIM, 0.1935 LPIPS, 0.0853 DISTS, 64.67 DOVER
RealisVideo-720P	28.23 PSNR, 0.8460 SSIM, 0.1414 LPIPS, 0.0844 DISTS, 77.34 DOVER
VideoLQ	51.96 DOVER, 56.40 MUSIQ, 5.18 NIQE
RealisVideo-4K	30.90 PSNR, 0.8746 SSIM, 0.0939 LPIPS, 0.1310 DISTS, 70.06 DOVER

On SPMCS, UDM10, REDS30, and YouTube-HQ, the method is reported to outperform prior methods on most metrics, with especially strong results on PSNR, SSIM, LPIPS, and DISTS (Zhao et al., 25 Jul 2025). On RealisVideo-720P, the paper emphasizes that the gains are notable because the benchmark is detail-rich despite being evaluated at 720P (Zhao et al., 25 Jul 2025). On VideoLQ, the authors caution that no-reference metrics can be misleading because artifacts may inflate them, although RealisVSR obtains the best DOVER score (Zhao et al., 25 Jul 2025).

The temporal-consistency analysis uses warping error $C=3$9. RealisVSR reports 0.81 on SPMCS, 1.72 on UDM10, 0.83 on REDS, and 1.59 on YouTube-HQ, all better than compared methods including RealViformer, VEnhancer, MGLD-VSR, and STAR (Zhao et al., 25 Jul 2025). The paper specifically states that on SPMCS this beats STAR’s 2.00 by 60% and RealViformer’s 1.32 by 37%, and on UDM10 improves over VEnhancer’s 2.47 by 25% (Zhao et al., 25 Jul 2025).

Qualitatively, the paper states that RealisVSR reconstructs skin texture, eyebrows, glasses, necklaces, poolside clutter, and shadows more faithfully than competing methods, which often either blur these details or generate inconsistent textures (Zhao et al., 25 Jul 2025). This is consistent with the stated role of HR-Loss as a dedicated mechanism for detail enhancement.

7. Interpretation, scope, and limitations

RealisVSR’s central thesis is that real-world high-resolution VSR requires a joint treatment of temporal consistency, artifact suppression, and high-frequency detail recovery, and that standard 720P benchmarks are insufficient to test this regime (Zhao et al., 25 Jul 2025). Its three-part design reflects that thesis directly: Wan2.1 provides the spatio-temporal prior, CPC targets conditioning stability and artifact suppression, and HR-Loss explicitly rectifies the tendency of diffusion objectives to prioritize low-frequency structure over fine texture (Zhao et al., 25 Jul 2025).

The benchmark contribution is inseparable from the modeling contribution. The introduction of RealisVideo-4K and RealisVideo-720P is not merely additive; it functions as an argument that evaluation at insufficient resolution obscures the very failures the method is designed to fix (Zhao et al., 25 Jul 2025). This suggests that the paper should be read simultaneously as a method paper and as a critique of prevailing VSR evaluation practice.

The stated limitations are practical rather than conceptual. The paper notes that some compared methods cannot run at full 4K due to GPU memory constraints and therefore must be evaluated via bicubic upscaling of lower-resolution outputs, complicating fair comparison (Zhao et al., 25 Jul 2025). It also acknowledges that no-reference metrics for real videos can be unreliable because artifact-heavy outputs may score artificially high (Zhao et al., 25 Jul 2025). On the efficiency side, although RealisVSR is faster than several diffusion-based competitors, the 480P version still requires four inference passes to reconstruct a final 720P video, so the computational cost remains nontrivial (Zhao et al., 25 Jul 2025).

Within the reported evidence, RealisVSR is therefore best understood as a detail-oriented real-world VSR system for ultra-high-resolution video restoration: a Wan2.1-based diffusion model whose main innovations are a ControlNet variant designed to preserve consistency, a loss formulation targeted at multiscale and directional high-frequency reconstruction, and a benchmark explicitly constructed to evaluate 4K detail recovery rather than only conventional 720P performance (Zhao et al., 25 Jul 2025).