SparkVSR: Interactive Video Super-Resolution

• SparkVSR is an interactive video super-resolution framework that uses sparse high-res keyframes as control signals to enhance low-resolution videos.
• It propagates refined details from edited or enhanced keyframes across the video using a diffusion-transformer mechanism, ensuring consistent temporal flow.
• The two-stage latent-pixel training and flexible keyframe guidance enable robust restoration, stylization, and applications in old-film processing.

SparkVSR is an interactive video super-resolution (VSR) framework that leverages sparse, high-resolution (HR) keyframes as a user- or algorithm-supplied control signal, enabling controllable and temporally consistent restoration of low-resolution (LR) video sequences. Rather than operating as a traditional black-box VSR model, SparkVSR supports direct human and algorithmic intervention by accepting edited or enhanced keyframes, propagating their details throughout the video via a dedicated propagation mechanism while adhering to the underlying LR video motion. Experiments demonstrate that SparkVSR achieves strong performance on both full-reference and no-reference VSR benchmarks, and generalizes without retraining to restoration and stylization tasks (Yu et al., 17 Mar 2026).

1. Motivation and Key Design Principles

Traditional VSR models do not admit user feedback during inference, leading to results that are not easily modified if artifacts or undesired details appear. Super-resolution is intrinsically ill-posed, with many plausible reconstructions per LR frame. User-desired features such as target textures or sharpness often cannot be specified directly. While single-image super-resolution (ISR) models—including state-of-the-art diffusion-based methods—deliver high-quality per-frame results, their application to video causes temporal inconsistencies, notably flicker.

SparkVSR is designed to break this limitation by introducing sparse keyframes—frames enhanced to HR by any ISR model or manually edited—as explicit control signals. These keyframes function as high-quality, editable anchors. SparkVSR then propagates the information from these anchors across the video, ensuring temporal consistency while grounding all changes in the true LR motion. The approach enables both fine-grained control and compatibility with arbitrary, modern ISR models.

2. Core Methodology: Keyframe-Conditioned Propagation

2.1 Keyframe Selection and Encoding

A sparse subset of frames (typically 1–4 per clip) is selected manually, by automatic policy (e.g., I-frame extraction from codecs), or randomly. Each selected frame is super-resolved using an external ISR model (e.g., Nano-Banana-Pro or PiSA-SR), producing HR reference images. SparkVSR encodes the full LR video $$x_{lr} \in \mathbb R^{T \times H \times W \times 3}$$ into a latent representation $Z_{LR}$ via a pretrained 3D causal variational autoencoder (VAE). The HR references for keyframes (at indices $\mathcal{K}$) are encoded as $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$ for $i \in \mathcal{K}$, set to zero elsewhere. The resulting $Z_{ref}$ is concatenated with $Z_{LR}$ channel-wise: $$Z_{in} = \operatorname{Concat}(Z_{LR}, Z_{ref}) \in \mathbb R^{\tfrac{T}{4} \times 32 \times H' \times W'}$$ This fusion allows for flexible integration of any off-the-shelf ISR output as a guide, enhancing generality.

2.2 Diffusion-Transformer Propagation

A one-step diffusion transformer $v_\theta$ (DiT), fixed at $t=399$ denoising steps, receives $Z_{LR}$0 and refines the latent $Z_{LR}$1 to $Z_{LR}$2, exploiting both the full LR context and the sparse HR references. The decoded $Z_{LR}$3 yields the final HR video $Z_{LR}$4. This architecture is robust to the absence or imperfection of keyframe references, as $Z_{LR}$5 can be zeroed and the model falls back to blind super-resolution behavior.

3. Training Pipeline: Latent-Pixel Two-Stage Optimization

3.1 Stage 1: Latent-Space Training

In the first stage, SparkVSR receives video latents ($Z_{LR}$6) and sparse reference latents ($Z_{LR}$7), with aggressive augmentations applied to simulate possible ISR artifacts. Reference dropout ($Z_{LR}$8) sets $Z_{LR}$9 at random, teaching the model both reference-conditioned and blind SR. A single transformer pass denoises from $\mathcal{K}$0 to $\mathcal{K}$1, supervised using mean squared error in latent space: $\mathcal{K}$2 This stage reinforces robust propagation of HR priors and ensures the network learns to interpolate or extrapolate from sparse references.

3.2 Stage 2: Pixel-Space Training

With the transformer and VAE fixed, outputs are decoded to pixel space for further refinement. Two parallel branches operate:

• Video branch: Receives video snippets with sparse references; outputs $\mathcal{K}$3. Supervised by pixelwise MSE, perceptual DISTS loss, and temporal frame-consistency loss using ground-truth or estimated optical flow. The aggregate loss:

$\mathcal{K}$4

with $\mathcal{K}$5.

• Image branch: Receives a single LR image latent and zeros for $\mathcal{K}$6, supporting strong blind SR prior learning when no references exist.

This two-stage strategy jointly increases perceptual fidelity (captured by DISTS, CLIP-IQA, and MUSIQ) and temporal consistency (DOVER, FasterVQA), while maintaining or improving full-reference PSNR/SSIM.

4. Inference and Guidance: Flexible User Control

4.1 Keyframe Selection Policies

Keyframes can be selected as follows:

• Manual selection: User directly specifies frames to guide propagation.
• Codec I-frame extraction: Intra-coded frames from compressed streams, usually of higher quality, serve as references.
• Random sampling: Enables automated large-scale or batch processing.

All approaches yield LR keyframe crops to be enhanced by the external ISR and then used as control signals.

4.2 Reference-Free Guidance Mechanism

To support variable reliance on keyframe information, SparkVSR leverages reference-dropout during training and a reference-free (classifier-free) guidance mechanism at inference. For each prediction: $\mathcal{K}$7 A guidance parameter $\mathcal{K}$8 modulates adherence to keyframe priors: $\mathcal{K}$9 Where:

• $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$0: keyframe-guided VSR (default).
• $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$1: strong keyframe adherence (less plausible in the presence of noisy references).
• $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$2: interpolated guidance.
• $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$3: blind SR.

This mechanism enables real-time balancing of restoration fidelity and adherence to user priors, supporting flexible adaptation to varying reference quality.

5. Empirical Evaluation and Ablation Studies

5.1 Datasets and Metrics

Experiments utilize diverse data:

• Synthetic: UDM10, SPMCS, YouHQ40 (with matched degradations).
• Authentic: RealVSR (paired smartphone clips), and the new MovieLQ (vintage film scan dataset).

Metrics include full-reference (PSNR, SSIM, LPIPS) and no-reference (CLIP-IQA, MUSIQ for image quality; FasterVQA, DOVER for video consistency).

5.2 Performance

On UDM10, SparkVSR with reference-free inference already matches or exceeds state-of-the-art blind diffusion VSR baselines (DOVE, STAR, SeedVR2, FlashVSR) on PSNR/SSIM. When Nano-Banana-Pro or PiSA-SR references are used:

• CLIP-IQA: +24.6% versus baseline
• DOVER: +21.8% versus baseline
• MUSIQ: +5.6% versus baseline

On MovieLQ, SparkVSR achieves MUSIQ = 68.88, CLIP-IQA = 0.636, FasterVQA = 0.803, and DOVER = 0.621 (Yu et al., 17 Mar 2026).

5.3 Ablations

• Stage-1 vs Stage-1+Stage-2: Pixel-space refinement adds significant perceptual improvements with minimal PSNR loss.
• Number of Keyframes: Even one reference frame yields a ~5-point boost in MUSIQ; additional keyframes bring diminishing returns but improve temporal smoothness.
• Guidance Parameter Sweep: Adjusting $Z_{ref}^{(i)} = \mathcal E_{sparse}(X_{ref}^{(i)})$4 traces a Pareto frontier between perceptual and distortion metrics, outperforming prior art in all regimes.
• Analysis of Temporal Consistency: X–T slice visualizations confirm recovery of sharp spatial features and suppression of temporal flicker.

6. Generalization and Applications

SparkVSR generalizes without retraining to novel tasks, as its inference pipeline is agnostic to the source of HR keyframes:

• Old-film restoration and colorization: User-edited or externally colorized/restored frames propagate clean textures and plausible color across an entire vintage clip.
• Stylized video generation: Applying a style transformation (e.g., anime) to sparse keyframes results in temporally consistent stylization across the video sequence, with the motion preserved.

These results highlight SparkVSR's utility as a general, interactive, keyframe-conditioned video processing platform, enabling a range of controllable applications in both restoration and creative domains.

7. Summary and Significance

SparkVSR introduces a new paradigm in VSR by directly incorporating sparse keyframes as user- or model-provided guides, enabled by a two-stage latent–pixel training regime and a diffusion transformer backbone. Its flexible integration with state-of-the-art ISR models, robust reference-free guidance mechanism, and strong empirical performance on both synthetic and real-world data demonstrate its potential as a generic, interactive VSR tool with broad applicability to restoration, stylization, and beyond (Yu et al., 17 Mar 2026).