Video Super-Resolution: Methods & Advances

• Video Super-Resolution (VSR) is the process of reconstructing high-resolution videos from low-resolution inputs by leveraging intra-frame spatial details and inter-frame temporal redundancy.
• Techniques vary from explicit optical flow and deformable convolutions to implicit feature matching and recurrent networks, each tailored for specific degradation models and application constraints.
• Recent advances integrate generative models, efficient training protocols, and hardware-aware designs to tackle challenges in perceptual quality, real-time processing, and deployment.

Video super-resolution (VSR) is the problem of reconstructing a high-resolution (HR) video from a low-resolution (LR) video by exploiting both intra-frame spatial structure and inter-frame temporal redundancy. In the survey formulation, if the $i$-th LR frame is $I_i \in \mathbb{R}^{H\times W\times 3}$, its HR counterpart is $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$, and degradation can be written as $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$, where $D$ is downsampling, $B$ is blur, $E_{i\rightarrow j}$ is warping, and $n_j$ is noise; the inverse problem is then to estimate $$\tilde I_i = \phi^{-1}(I_i,\{I_j\}_{j=i-N}^{i+N};\theta_\beta)$$ from a temporal neighborhood (Liu et al., 2020). A common supervised formulation uses a centered LR window $X_{t-L:t+L}$ to predict an HR target frame $I_i \in \mathbb{R}^{H\times W\times 3}$0, which remains the dominant template even when the application domain is specialized, such as video face super-resolution (Xin et al., 2020).

1. Problem formulation and degradation models

The central distinction between VSR and single-image super-resolution is the use of inter-frame information. The 2020 survey emphasizes that VSR is fundamentally organized by how methods exploit neighboring frames, because temporal evidence can reveal sub-pixel shifts and complementary details absent from the target LR frame (Liu et al., 2020). In practice, this leads to several related problem settings rather than a single canonical task.

A standard setting is multi-frame center-frame reconstruction: a short LR clip is provided, and the network reconstructs the HR version of the middle frame. This appears explicitly in face-centric VSR, where the input is a temporal window $I_i \in \mathbb{R}^{H\times W\times 3}$1 and the output is $I_i \in \mathbb{R}^{H\times W\times 3}$2; the specialization lies not in the formal task, but in the motion distribution, which includes rigid head motion, facial expression changes, and background motion around cropped faces (Xin et al., 2020). Other methods instead predict an entire HR sequence from an LR sequence, either with fixed clips or recurrent processing.

The degradation model varies substantially across subfields. Synthetic bicubic downsampling remains common in benchmark-oriented work and challenge setups, including REDS-based mobile VSR and much of the benchmark literature summarized in the survey (Liu et al., 2020). More structured synthetic degradations are also used. In video face super-resolution, LR inputs are generated by blur, bicubic downsampling, and additive Gaussian noise, with the paper writing $I_i \in \mathbb{R}^{H\times W\times 3}$3 and instantiating blur by a $I_i \in \mathbb{R}^{H\times W\times 3}$4 Gaussian kernel with standard deviation $I_i \in \mathbb{R}^{H\times W\times 3}$5, scale factor $I_i \in \mathbb{R}^{H\times W\times 3}$6, and Gaussian noise level $I_i \in \mathbb{R}^{H\times W\times 3}$7 (Xin et al., 2020). In compressed VSR, the LR input is additionally passed through a codec; VSR-HE explicitly targets decoder-side restoration of HEVC-compressed LR streams across QP values $I_i \in \mathbb{R}^{H\times W\times 3}$8, turning VSR into a compound artifact-removal and upscaling problem (Jiang et al., 17 Jun 2025).

Causal online VSR introduces a further restriction: only the current LR frame and previous frames are available. TS-Mamba formulates this setting as reconstructing $I_i \in \mathbb{R}^{H\times W\times 3}$9 from $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$0 and $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$1, in contrast to offline methods that can exploit both past and future context (Zhu et al., 14 Aug 2025). This suggests that the VSR literature is best understood as a family of inverse problems whose defining variables are not only scale factor and degradation, but also temporal access, motion assumptions, and deployment constraints.

2. Taxonomy of temporal information utilization

A widely used organizational principle is the survey taxonomy based on how inter-frame information is used. The survey divides deep VSR into methods with alignment and methods without alignment, and refines them into seven sub-categories: motion estimation and motion compensation (MEMC), deformable convolution (DC), 2D convolution, 3D convolution, recurrent convolutional neural network (RCNN), non-local, and other methods (Liu et al., 2020).

Category	Core mechanism	Representative examples
MEMC	explicit motion estimation, warping, then fusion	VESPCN, FRVSR, TOFlow, BasicVSR, IconVSR
Deformable convolution	learned offsets for alignment	TDAN, EDVR, VESR-Net
2D convolution	implicit temporal fusion with stacked frames	VSRResFeatGAN, FFCVSR
3D convolution	joint spatio-temporal filtering	DUF, 3DSRnet, DSMC
RCNN	hidden-state propagation across time	BRCN, RLSP, RSDN
Non-local	global pairwise correlation modeling	PFNL
Other	hybrid or back-projection-style designs	RBPN, STARnet, DNSTNet

This taxonomy remains useful for later architectures, although post-2020 work often combines elements that were previously separate. VSR-Transformer replaces both standard Transformer sublayers with VSR-specific modules—spatial-temporal convolutional self-attention and bidirectional optical-flow-based feed-forward—so it lies at the intersection of attention, explicit alignment, and fixed-clip processing rather than fitting neatly into a single survey-era class (Cao et al., 2021). SATVSR likewise uses optical flow, but not primarily for warping; instead it uses optical-flow-guided patch labels to constrain transformer attention to same-label candidates, making temporal aggregation selective under cross-scene transitions (Chen et al., 2022).

A plausible implication is that the historical sequence of VSR design has moved from explicit pairwise alignment, toward feature-space alignment and recurrent propagation, and then toward sparse or selective long-range aggregation. The survey already noted a shift from local sliding windows toward broader temporal dependency modeling (Liu et al., 2020), and later online state-space and sparse-attention designs make that trend explicit (Zhu et al., 14 Aug 2025).

3. Alignment, propagation, and feature interaction

The dominant technical question in VSR is how motion should be represented and exploited. Classical deep VSR often estimates optical flow and warps neighboring frames or features before fusion. This remains explicit in VSR-Transformer, whose BOFF layer estimates bidirectional optical flow, warps features, and performs forward/backward propagation before fusion; the paper argues that a standard token-wise feed-forward layer is inadequate because it lacks cross-frame interaction and alignment (Cao et al., 2021). Fast Online VSR with Deformable Attention Pyramid keeps a recurrent hidden state and uses a Deformable Attention Pyramid (DAP) to predict only a limited number of spatial sampling locations, with $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$2, thereby aligning and integrating previous hidden-state information under strict causality (Fuoli et al., 2022).

A second line avoids explicit flow-and-warp pipelines and instead models motion implicitly in feature space. The Motion-Adaptive Feedback Network for video face super-resolution is explicit on this point: it contains no optical flow estimation, no image warping, no spatial transformer-based alignment, no deformable alignment, and no attention mechanism in the modern sense. Instead, its Motion-Adaptive Feedback Cell compares features from adjacent frames at the same depth and injects motion-adaptive signals back into later feature refinements. At layer $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$3, frame $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$4 is updated as $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$5, preserving per-frame streams while adding pairwise motion descriptors (Xin et al., 2020). DSMC similarly avoids explicit optical flow in its main temporal reasoning, using a U-shaped residual dense network with 3D convolution for fine implicit motion estimation and motion compensation, together with Multi-Stage Communicated Upsampling to re-inject intermediate image estimates into feature refinement (Liu et al., 2021).

A third line replaces dense matching by selective or constrained matching. SATVSR restricts attention to patches that share optical-flow-guided labels and then selects the top-1 most similar patch to supplement the target representation, aiming to suppress irrelevant temporal information in fast scene switching or cross-scene clips (Chen et al., 2022). TS-Mamba adopts a causal variant of this idea: it constructs trajectories across previous frames, computes cosine similarity between current tokens and trajectory-linked previous tokens, keeps the top-$\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$6 most similar tokens with $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$7, and aggregates them using Trajectory-aware Shifted Mamba Aggregation (Zhu et al., 14 Aug 2025). This suggests an increasingly common design principle: temporal utilization is becoming more selective, either because explicit alignment is brittle or because all-to-all aggregation is too expensive.

4. Architectural regimes and deployment-oriented specializations

VSR architectures now span several distinct operational regimes. Fixed-clip feed-forward models remain common. GRRN uses 7 consecutive LR frames, a 3D spatio-temporal front-end, and a grouped residual-in-residual body with grouped point-wise and depth-wise convolutions; it is explicitly non-recurrent and avoids optical flow, deformable alignment, and recurrent propagation (Ashoori et al., 2023). PP-MSVSR instead combines short-range and long-range processing in three stages: a Local Fusion Module aligns $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$8 features before propagation, stage-2 uses a BasicVSR++-style bidirectional propagation structure with auxiliary supervision, and stage-3 reuses offsets and masks from stage-2 in a Re-Align Module (Jiang et al., 2021).

Online VSR introduces a separate architecture family built around recurrence and latency. DAP is explicitly causal and real-time, carrying a hidden state $\hat I_i \in \mathbb{R}^{sH\times sW\times 3}$9 forward while using sparse deformable attention to align it to the current frame (Fuoli et al., 2022). TS-Mamba generalizes online temporal access beyond one previous frame by retrieving multiple past tokens along estimated trajectories and aggregating them with state space models; on $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$0 LR inputs it reports 112G MACs, 3.0M parameters, 29 ms runtime, and 33.5 FPS (Zhu et al., 14 Aug 2025).

Some recent work is specialized by degradation or deployment rather than by temporal model. VSR-HE targets HEVC-compressed content for video conferencing, but under challenge rules it processes each frame independently, so it is better understood as a compression-aware single-frame $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$1 SR model applied to video frames rather than a temporal VSR backbone (Jiang et al., 17 Jun 2025). RepNet-VSR is optimized for mobile NPU deployment: it takes 10 LR frames packed into 30 channels, uses ordinary convolutions with a reparameterizable bottleneck block, and reports 27.79 dB PSNR on the REDS validation set for $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$2 at 103 ms per 10 frames on a MediaTek Dimensity NPU (Wu et al., 22 Apr 2025).

Generative VSR introduces yet another regime. TurboVSR treats super-resolution as conditional latent generation, compresses videos with an autoencoder at a factor of $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$3, factorizes conditioning into first-frame SR followed by conditioned video SR, and converts a pretrained flow-matching diffusion model into a shortcut model for 4-step sampling. It reports 7 seconds for a 2-second 1080p video and a speedup of $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$4 relative to prior diffusion-based VSR methods while remaining competitive in perceptual quality (Wang et al., 30 Jun 2025). This suggests that recent VSR research is no longer dominated by a single backbone family; instead, architecture is strongly conditioned by whether the target is benchmark fidelity, online causality, compressed content, mobile deployment, or generative detail synthesis.

5. Training protocols, evaluation, and efficiency

Benchmark practice in VSR remains heterogeneous. The survey identifies Vid4 as the most popular test set, Vimeo-90K as the most widely used training set, and REDS/REDS4 as a challenging large-motion benchmark (Liu et al., 2020). Later papers preserve this pattern but add domain-specific datasets: face VSR uses VoxCeleb with 3884 training sequences, 10 validation sequences, and 697 test sequences (Xin et al., 2020); compressed VSR challenge work uses REDS plus BVI-AOM for Track 1 and VCD plus BVI-AOM for Track 2 (Jiang et al., 17 Jun 2025).

Evaluation metrics vary with task assumptions. The survey foregrounds PSNR and SSIM and notes that results are often reported on the Y channel for Vid4 and Vimeo-90K-T, but on RGB for REDS4 in some comparisons (Liu et al., 2020). Compression-aware VSR extends this to PSNR-Y, SSIM, MS-SSIM, and VMAF (Jiang et al., 17 Jun 2025). Diffusion-based real-world VSR emphasizes perceptual and no-reference measures such as DOVER, MUSIQ, and NIQE, precisely because PSNR and SSIM may overreward blurrier outputs in ill-posed restoration (Wang et al., 30 Jun 2025). This suggests that metric selection is inseparable from the degradation model and the restoration objective.

Efficiency has become a first-class research topic. “Accelerating the Training of Video Super-Resolution Models” shows that VSR training can be sped up by an easy-to-hard multigrid schedule over spatial crop size and temporal window length, combined with large-minibatch training, yielding up to $I_j = D B E_{i\rightarrow j}\hat I_i + n_j$5 speedup in wall-clock training time without performance drop on BasicVSR-M (Lin et al., 2022). On the inference side, deployment-driven papers now report latency, FLOPs, and hardware. DAP-128 reports 38 ms/frame and 26.3 fps on an RTX2080Ti while surpassing EDVR-M on REDS4 and UDM10 (Fuoli et al., 2022). TS-Mamba reports the lowest MACs among listed online baselines at 112G (Zhu et al., 14 Aug 2025). RepNet-VSR and TurboVSR treat hardware throughput as a primary design constraint rather than a secondary comparison point (Wu et al., 22 Apr 2025).

The survey also warns that direct numerical comparison across papers is imperfect because methods differ in degradation assumptions, input-frame counts, crop conventions, channels used for evaluation, and whether all test frames are utilized (Liu et al., 2020). That caveat remains accurate for later work, where synthetic blur-downsampling, bicubic degradation, HEVC compression, and real-world degradations coexist in the same literature.

6. Limitations, controversies, and open problems

Several unresolved issues recur across the literature. The survey highlights lightweight models, interpretability, larger and arbitrary scale factors, more reasonable degradation models, unsupervised VSR, scene-change handling, and better evaluation criteria as open challenges (Liu et al., 2020). Many later papers instantiate these limitations rather than resolving them completely.

One persistent issue is degradation realism. Face VSR with motion-adaptive feedback is evaluated only on synthetic LR videos generated from VoxCeleb, so robustness to real LR face videos is unproven (Xin et al., 2020). Compressed VSR-HE is codec-specific to HEVC/H.265 and remains frame-independent, with no temporal fusion or temporal consistency loss (Jiang et al., 17 Jun 2025). UVSR, the deep unrolled model, assumes known blur and downsampling and is explicitly nonblind, which improves interpretability but restricts practical scope (Chiche et al., 2021).

A second issue is the tension between efficiency and temporal sophistication. Training acceleration via multigrid was demonstrated only for CNN-based VSR, and its applicability to Transformer-based VSR was left as future work (Lin et al., 2022). TS-Mamba improves the online accuracy-efficiency trade-off, but its token selection depends on trajectory quality, and severe occlusion or inaccurate flow could degrade retrieval (Zhu et al., 14 Aug 2025). RepNet-VSR achieves mobile efficiency partly by avoiding explicit temporal alignment, which plausibly limits its robustness to large or non-rigid motion (Wu et al., 22 Apr 2025).

A third issue concerns perceptual realism versus fidelity metrics. TurboVSR shows that diffusion-based VSR can produce strong perceptual detail at dramatically lower runtime than earlier diffusion baselines, but the paper also notes weaknesses of the underlying highly compressed latent representation on fast motion and repetitive regular textures, and does not yet demonstrate 4K video SR (Wang et al., 30 Jun 2025). This suggests that high-fidelity generative VSR remains constrained by temporal stability, token efficiency, and video-scale training data.

A final, longstanding controversy is how best to exploit neighboring frames. Some work argues that explicit motion compensation is brittle under complex motion and that feature-space alternatives are preferable (Xin et al., 2020); other work shows that carefully designed optical-flow-guided or deformable alignment remains highly effective (Cao et al., 2021). The survey’s own synthesis implies that no single mechanism is universally dominant: alignment, recurrence, deformable sampling, non-locality, sparse attention, and generative priors all solve different parts of the same inverse problem (Liu et al., 2020). The field’s current trajectory suggests not convergence to one architecture, but increasingly task-specific combinations of temporal selection, spatial reconstruction, and hardware-aware design.