Rethinking Alignment in Video Super-Resolution Transformers (2207.08494v2)

Abstract: The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. These observations indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select alignment methods for different networks/datasets. Codes and models will be released at https://github.com/XPixelGroup/RethinkVSRAlignment.

• The paper challenges traditional alignment in VSR by demonstrating that Transformer self-attention effectively aggregates multi-frame information without explicit alignment.
• It compares various alignment strategies on REDS4 and Vimeo-90K, revealing that patch-level or no alignment can achieve competitive PSNR and SSIM scores.
• The proposed Patch Alignment technique sidesteps optical flow issues by aligning image patches, paving the way for more efficient and robust VSR models.

Analysis of Alignment in Video Super-Resolution Transformers

The paper presents a rigorous analysis of the role of alignment in Video Super-Resolution (VSR) using Transformer architectures. VSR aims to enhance the resolution of video frames by exploiting information from multiple low-resolution frames. Traditional approaches to VSR heavily rely on alignment modules to compensate for inter-frame motion, which are computationally intensive and may inadvertently degrade performance. This paper challenges conventional wisdom by proposing a reassessment of alignment strategies in the context of Transformers, a paradigm recently adopted for VSR tasks.

Key Observations and Methodology

Two core observations are central to this work:

1. Direct Utilization of Information: VSR Transformers can harness multi-frame information from unaligned video frames. This capability is attributed to the self-attention mechanism of Transformers, which effectively models long-range dependencies without explicit alignment.
2. Detrimental Effects of Alignment: Incorporating traditional alignment methods, such as optical flow-based methods, can sometimes hinder the performance of VSR Transformers by corrupting sub-pixel information essential for reconstructing high-resolution frames.

The paper employs various alignment configurations, including image alignment, feature alignment, and no alignment, across two prominent VSR benchmarks: REDS4 and Vimeo-90K. These datasets present different challenges in terms of object motion, with REDS4 featuring more substantial movement that typically necessitates alignment for accurate super-resolution.

Key Findings and Numerical Results

The research presents convincing evidence that enlarging the Transformer's attention window allows for an alignment-free operation that retains or even improves VSR performance for smaller degrees of motion:

• For small motion datasets like Vimeo-90K, the absence of alignment results in comparable or improved performance over methods with alignment, yielding PSNR and SSIM scores of 37.46 and 0.9474, respectively.
• On the REDS4 dataset, which includes larger motions, the proposed Patch Alignment method outperforms traditional alignment approaches with a PSNR of 31.11, showcasing the superiority of patch-level processing over pixel-level alignment in large-motion contexts.

Patch Alignment: A Novel Approach

To address the computational burden and limitations of traditional alignment, the paper introduces Patch Alignment, a technique focusing on aligning patches rather than individual pixels. By considering image patches as atomic units, this method circumvents the destructive effects of pixel misalignment and optical flow inaccuracies. Patch Alignment is implemented using a nearest-neighbor approach to maintain intra-patch pixel relationships, crucial for retaining aliasing patterns informative for reconstruction.

Implications and Future Directions

The implications of this research are far-reaching for the development of VSR technologies. By demonstrating that Transformers can effectively process unaligned frames, the paper opens avenues for more efficient, simpler VSR models devoid of complex alignment modules. This paradigm shift may foster new architectures in video restoration, where the Transformer's native capabilities are leveraged to their full potential.

Future exploration could focus on extending these findings to other video restoration tasks and developing even more specialized Transformer architectures. The patch-based alignment strategy introduced here could be further optimized for cases involving non-uniform motion patterns, offering a unified framework adaptable across diverse video content domains. Thus, this research lays a foundation for reimagining video super-resolution in an era increasingly dominated by Transformer models.