Rethinking Image Super-Resolution from Training Data Perspectives

Abstract: In this work, we investigate the understudied effect of the training data used for image super-resolution (SR). Most commonly, novel SR methods are developed and benchmarked on common training datasets such as DIV2K and DF2K. However, we investigate and rethink the training data from the perspectives of diversity and quality, {thereby addressing the question of ``How important is SR training for SR models?''}. To this end, we propose an automated image evaluation pipeline. With this, we stratify existing high-resolution image datasets and larger-scale image datasets such as ImageNet and PASS to compare their performances. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance. We hope that the proposed simple-yet-effective dataset curation pipeline will inform the construction of SR datasets in the future and yield overall better models.

• The paper introduces a novel training data perspective by showcasing how curated low-resolution images with diverse object regions can outperform traditional high-resolution datasets.
• The paper proposes an automated image evaluation pipeline that filters low-quality images and retains those with abundant object details to enhance dataset robustness.
• The paper demonstrates that SR models trained on the DiverSeg dataset achieve higher PSNR and SSIM scores across multiple architectures, validating the approach's effectiveness.

Rethinking Image Super-Resolution from Training Data Perspectives

The paper "Rethinking Image Super-Resolution from Training Data Perspectives" by G. Ohtani et al. presents an in-depth analysis of how training data influences the performance of image super-resolution (SR) models. It escapes the traditional focus on solely improving neural network architectures and instead scrutinizes the quality and diversity of the training datasets employed.

Key Methodology and Contributions

The pivotal contribution of this work is its introduction of the Diverse Segmentation dataset (DiverSeg), composed of low-resolution yet high-quality images with a significant diversity of object regions. The research team challenges the prevailing perspective that high-resolution images are indispensable for training SR models.

Automated Image Evaluation Pipeline

The authors developed an automated image evaluation pipeline aimed at curating datasets based on two primary criteria: resolution and diversity. This pipeline is bifurcated into two key processes:

1. Source Selection: This process estimates the quality of various low-resolution image datasets (e.g., ImageNet-1k, PASS) by utilizing a blockiness measure. Datasets with estimated quality below a threshold are filtered out, ensuring that the SR models are trained on high-quality images.
2. Object-based Filtering: This method assesses the diversity of images by counting the number of object regions using advanced models for segmentation and detection. Images that do not meet a predefined threshold of object regions are discarded, enhancing the diversity and robustness of the resulting dataset.

Evaluation and Results

The empirical evaluation substantiates the effectiveness of the proposed approach. By comparing SR models trained on DiverSeg with those trained on conventional high-resolution datasets (DF2K and LSDIR), the authors demonstrate that the former can achieve superior performance.

Key Findings

1. Impact of Training Data Quality: SR models trained on low-quality images exhibit poor performance, evidenced by the appearance of artifacts such as stripes and checkerboard patterns. Therefore, the exclusion of low-quality images is critical.
2. Effectiveness of Diverse Object Regions: Incorporating images with a high number of object regions significantly enhances SR model performance. This conclusion is derived from comparative analyses between various filtering techniques, where segmentation-based filtering consistently outperformed other methods.
3. Superior Dataset Performance: DiverSeg datasets not only surpassed traditional high-resolution datasets in terms of PSNR and SSIM metrics but also streamlined the training process by reducing data redundancy. This performance reinforcement was consistent across multiple SR models, including MSRResNet, EDSR, RCAN, SwinIR, and HAT.

Implications and Future Work

The implications of this study are multifaceted. Practically, the automated pipeline provides an efficient tool for curating high-quality and diverse datasets from large collections of low-resolution images. Theoretically, the findings encourage a paradigm shift, suggesting that the quality and diversity of training data are as crucial as the ingenuity of model architectures.

Succeeding investigations could extend this work by exploring hybrid datasets that integrate both low- and high-resolution images while leveraging the filtering processes detailed in this paper. Additionally, the methodology could be adapted for blind super-resolution tasks where degradation processes are not explicit, providing broader applicability in real-world scenarios.

Overall, this paper enriches the SR community by underlining the significance of training data perspectives and furnishing a robust methodology for dataset curation that enhances model performance independent of image resolution restraints.