Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes (2104.01601v1)

Abstract: Joint rolling shutter correction and deblurring (RSCD) techniques are critical for the prevalent CMOS cameras. However, current approaches are still based on conventional energy optimization and are developed for static scenes. To enable learning-based approaches to address real-world RSCD problem, we contribute the first dataset, BS-RSCD, which includes both ego-motion and object-motion in dynamic scenes. Real distorted and blurry videos with corresponding ground truth are recorded simultaneously via a beam-splitter-based acquisition system. Since direct application of existing individual rolling shutter correction (RSC) or global shutter deblurring (GSD) methods on RSCD leads to undesirable results due to inherent flaws in the network architecture, we further present the first learning-based model (JCD) for RSCD. The key idea is that we adopt bi-directional warping streams for displacement compensation, while also preserving the non-warped deblurring stream for details restoration. The experimental results demonstrate that JCD achieves state-of-the-art performance on the realistic RSCD dataset (BS-RSCD) and the synthetic RSC dataset (Fastec-RS). The dataset and code are available at https://github.com/zzh-tech/RSCD.

• The paper’s main contribution is a unified approach that jointly addresses rolling shutter distortion and motion blur in dynamic scenes.
• It introduces the BS-RSCD dataset using a beam-splitter system to capture both distorted and clear references in realistic urban settings.
• The proposed JCD network leverages bi-directional warping and deformable attention, outperforming existing methods in metrics like PSNR, SSIM, and LPIPS.

A Comprehensive Examination of Joint Rolling Shutter Correction and Deblurring in Dynamic Scenes

In the field of computational photography and video processing, the paper "Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes" authored by Zhihang Zhong, Yinqiang Zheng, and Imari Sato presents notable contributions in addressing the challenges posed by rolling shutter distortion and motion blur, particularly prevalent in CMOS cameras widely used in consumer electronics. The paper introduces a novel dataset and a neural network architecture aimed at improving the effectiveness of rolling shutter correction and deblurring in dynamically changing environments.

The authors identify a crucial issue with prevalent techniques that separate rolling shutter correction (RSC) and deblurring into distinct problems, often based on assumptions that do not hold in real-world scenarios. Existing methods struggle when encountering images where both rolling shutter distortion and motion blur are present due to the motion of both the camera and scene objects. This intrinsic complexity stems from the need to address displacements and blur kernels simultaneously, an area inadequately explored in prior research.

To address these limitations, the authors make two primary contributions:

1. BS-RSCD Dataset: The paper emphasizes the lack of realistic datasets for the joint rolling shutter correction and deblurring (RSCD) problem. To tackle this, the authors introduce the BS-RSCD dataset, captured using a sophisticated beam-splitter acquisition system. This dataset includes videos captured with both rolling shutter and global shutter cameras to provide distorted/blurry and sharp references, respectively. The dataset is captured in dynamic urban environments with both ego-motion and object-motion, offering a crucial resource for developing and benchmarking RSCD solutions.
2. Joint Correction and Deblurring Model (JCD): The authors propose a novel neural network architecture that incorporates features from state-of-the-art approaches in both RSC and GSD. The architecture employs bi-directional warping streams alongside a deblurring stream, effectively addressing displacement and blur challenges, improved by a deformable attention module. This innovation allows the proposed model to achieve superior performance on BS-RSCD, demonstrating its potential in generating undistorted, sharp images from rolling shutter distorted inputs.

The paper includes robust experimental evaluations where JCD outperforms existing models on the BS-RSCD dataset as well as the Fastec-RS dataset, showcasing its versatility and effectiveness. The performance is quantified not only using standard metrics such as PSNR and SSIM but also the LPIPS metric, which better captures perceptual quality.

The implications of this research extend to various practical applications, including enhanced video quality for consumer devices using CMOS sensors, improved accuracy in computer vision tasks relying on video input, and new possibilities for capturing high-quality imagery in dynamic environments without the constraints of global shutter cameras.

In speculating future developments in AI and computational photography, the methods introduced in this paper pave the way for more integrated approaches in handling complex video degradation issues. It suggests a trend towards leveraging deep learning architectures that simultaneously address multiple facets of image and video restoration, moving beyond the traditional separation of tasks. Further developments could explore scalability in terms of handling higher resolution videos and real-time processing capabilities, as well as extending the dataset and methodology to other sensor types and configurations.

This paper firmly establishes a foundation for advancing technologies that improve visual media quality in practical, consumer-oriented contexts, reaffirming the continuing need for multidisciplinary approaches in the evolving landscape of image processing.