ISRGen-QA: Modern SR-IQA Benchmark

• ISRGen-QA is a modern SR-IQA dataset that evaluates super-resolved images produced by advanced generative methods (GAN and diffusion) with human MOS ratings.
• The dataset comprises 720 images (~2K resolution) derived from 19 HR references and processed by 15 diverse SR algorithms spanning multiple architectures.
• It supports both reference-based and blind quality assessments, serving as the foundation for the VQualA 2025 challenge with evaluation metrics like SRCC and PLCC.

ISRGen-QA, short for Image Super-Resolution Generated Content Quality Assessment, is a super-resolution image quality assessment dataset introduced as the foundation of the VQualA 2025 Challenge on Image Super-Resolution Generated Content Quality Assessment (ISRGC-Q). It is designed to evaluate perceptual quality on super-resolved images produced by contemporary generative super-resolution pipelines, especially GAN-based and diffusion-based methods, rather than the older interpolation- or early CNN-centered outputs that dominate earlier SR-IQA resources. The dataset contains 720 super-resolved images at approximately 2K resolution, derived from 19 high-resolution reference images and 76 low-resolution reference images, with human perceptual judgments aggregated as MOS and used as the target for quality prediction (Li et al., 8 Sep 2025).

1. Motivation and positioning within SR-IQA

ISRGen-QA was created in response to a specific shift in the super-resolution literature: modern SR systems increasingly synthesize texture and structure through strong generative priors, so perceptual quality can no longer be treated primarily as a problem of classical fidelity to a reference. The motivating paper frames SR as an ill-posed inverse problem in which multiple plausible high-resolution reconstructions may correspond to a single low-resolution input. In that setting, perceptual failure modes include not only blur, noise, and compression, but also over-sharpened edges, hallucinated or false textures, unnatural reconstruction patterns, texture inconsistencies, and edge distortions, together with the broader fidelity–naturalness tradeoff (Li et al., 8 Sep 2025).

The dataset is explicitly positioned against earlier SR-IQA collections such as QADS, Waterloo, SISR-IQA, CVIU, RealSRQ, and SISAR. Those resources are described as being dominated by interpolation methods, early learning-based approaches, or older CNN SR models, with only limited representation of modern generative SR. The paper notes that in prior datasets the newest included methods are around 2019–2020, and that SRGAN is essentially the only GAN-based representative. ISRGen-QA therefore functions as a benchmark update: it aims to make SR-IQA reflect the artifact regimes of current generative SR rather than legacy restoration pipelines.

2. Construction and composition

ISRGen-QA contains 720 super-resolved images at image sizes in the range $2040 \times 1152 \sim 2040 \times 1440$. The source material consists of 19 high-resolution reference images selected from DIV2K. These are converted into 76 low-resolution reference images by bicubic downsampling at four scales: $\times 2$, $\times 3$, $\times 4$, and $\times 8$. The low-resolution references are then processed by 15 advanced SR algorithms, producing the final SR image set (Li et al., 8 Sep 2025).

The included SR methods span several architectural families:

Family	Methods
GAN-based	ESRGAN, Real-ESRGAN, BSRGAN, SeD
Diffusion-based	SR3, IDM, SRDiff, CDFormer, SAM-DiffSR
Transformer-based	SRNO, ATD-SR, SwinIR, CAMixerSR
Flow-based	BFSR
CNN-based	EDSR

This construction deliberately mixes GAN-based, diffusion-based, transformer-based, flow-based, and CNN-based systems. The intent is not only architectural breadth, but also distortion diversity. The paper states that the reported total of 720 SR outputs does not come with a finer published breakdown by algorithm or scale; similarly, it does not specify how many samples each method contributes beyond the total count and the fact that all four scales are covered. This suggests that the benchmark is primarily defined by aggregate coverage rather than by a fully enumerated factorial design.

The dataset is partitioned using an 80/10/10 split into 576 training images, 72 validation images, and 72 test images. The paper presents this split as the standard experimental basis for reproducible model development and challenge evaluation (Li et al., 8 Sep 2025).

3. Subjective annotation and target labels

The target labels in ISRGen-QA are MOS values derived from human opinion scores. Subjective annotation involved 23 human participants, consisting of 11 female and 12 male raters from 5 different countries and of various ages. After anomaly filtering, the final valid scores were taken from 21 participants, and the challenge paper consistently treats the resulting MOS as the prediction target (Li et al., 8 Sep 2025).

The annotation protocol, however, is only partially documented. The paper does not specify the exact viewing setup, display conditions, presentation method, rating scale labels, whether the design was single-stimulus or double-stimulus, or the anomaly filtering criterion. It also does not define MOS with an explicit formula in the dataset description, and DMOS is not used. Likewise, the paper does not report label-distribution histograms, inter-rater agreement coefficients, confidence intervals, or score variances. As a result, ISRGen-QA is well specified as a benchmark corpus with human perceptual targets, but less fully specified as a psychophysical dataset than some dedicated IQA studies.

A common misconception is that ISRGen-QA is primarily a no-reference dataset because several strong submissions emphasize blind assessment. The paper does not support that characterization. During challenge development and testing, participants receive SR images together with their corresponding HR and LR reference images, and the benchmark is therefore best described as open with respect to reference usage rather than restricted to a single FR, RR, or NR regime.

4. Challenge protocol and evaluation framework

ISRGen-QA is operationalized through the ISRGC-Q task in the VQualA 2025 Challenge, which asks participants to predict perceptual quality scores for SR images so that predictions align with human MOS. The official evaluation metrics are Spearman rank-order correlation coefficient (SRCC) and Pearson linear correlation coefficient (PLCC). Final ranking uses the weighted score

$$\mathrm{Score} = 0.6 \times \mathrm{SRCC} + 0.4 \times \mathrm{PLCC}.$$

An ideal method is therefore one that is simultaneously strong in monotonic ranking fidelity and linear score regression (Li et al., 8 Sep 2025).

The challenge had two phases. In the development phase, participants received the 576 training SR images with MOS labels, along with the corresponding HR and LR references, and also received 72 validation SR images without MOS labels. In the testing phase, they received 72 test SR images and their HR and LR references, and final ranking was computed using the weighted score above. The challenge registered 108 participants overall, received 193 submissions from 12 teams during development, and obtained 5 final test submissions, of which 4 were retained in the official ranking because only those four teams submitted the required fact sheets and source codes.

The benchmark is methodologically notable because it accommodates different input regimes. The winning MICV method used only the SR image; QA-Veteran explicitly pursued a blind/no-reference approach; ydy used both SR and LR; and 2077 Agent used SR and HR. This makes ISRGen-QA a benchmark for perceptual prediction under heterogeneous reference assumptions rather than a benchmark tied to one canonical IQA setting.

5. Empirical behavior and challenge results

The official test-set leaderboard shows uniformly high performance but also persistent difficulty in fine-grained ranking. The final results were: MICV with overall 0.9638, ydy with 0.9429, QA-Veteran with 0.9409, and 2077 Agent with 0.9248. Their corresponding PLCC values ranged from 0.9476 to 0.9714, while SRCC ranged from 0.9096 to 0.9588 (Li et al., 8 Sep 2025).

A central empirical observation in the paper is that PLCC is consistently higher than SRCC for the top methods. The authors interpret this as evidence that score regression is easier than preserving perfect perceptual rank ordering across the full quality range, especially when images have similar perceptual quality. This makes ISRGen-QA non-trivial despite the high absolute scores: top systems can learn the global MOS mapping well, yet still struggle with local ordering consistency.

The methods themselves also indicate the range of modeling priors that ISRGen-QA can support. MICV uses a hybrid Vision Transformer + CNN and only the SR image; ydy combines ResNet50 backbones, BLIP-2 semantic features, SR-LR residual guidance, and a Mixture-of-Experts mechanism; QA-Veteran formulates blind SR-IQA as a resolution-adaptive vision-language problem using SigLIP2-NaFlex; and 2077 Agent emphasizes ultra-high-resolution perception with candidate-region sampling, ConvNeXt-Base, FPN, and histogram prediction over discrete quality levels. This variety suggests that ISRGen-QA is compatible with global-context models, blind IQA formulations, residual-reference methods, and region-aware perceptual aggregation.

6. Significance, limitations, and scope

ISRGen-QA is significant because it updates SR-IQA benchmarking to match the perceptual behavior of generated super-resolution content, especially outputs from GANs and diffusion models. Its design foregrounds artifact classes that are weakly represented in older SR-IQA datasets, including hallucinated textures, false detail, unnatural structures, and edge anomalies, and thereby reorients quality assessment toward the perceptual regimes of current SR research (Li et al., 8 Sep 2025).

At the same time, the challenge paper leaves several dataset-analysis questions open. It does not provide a detailed subjective-study protocol, label-distribution analysis, inter-rater agreement, per-distortion breakdowns, per-model-family breakdowns, or per-scale breakdowns. It also does not present baseline results from prior IQA literature on the dataset itself, so the main comparative evidence comes from the four final challenge teams. Nor does it provide a dedicated empirical artifact-by-artifact comparison between GAN-generated and diffusion-generated SR outputs; that distinction appears primarily as motivation rather than as a reported analysis result.

The public-facing role of ISRGen-QA is therefore twofold. First, it is a modern SR-IQA dataset built around contemporary generative SR rather than legacy SR outputs. Second, it is a challenge benchmark that exposes an open methodological question: whether perceptual quality prediction for generated SR is best approached by blind IQA, reference-assisted quality assessment, multimodal semantic modeling, or region-aware ultra-high-resolution analysis. The project is reported as publicly available through the ISRGen-QA repository at https://github.com/Lighting-YXLI/ISRGen-QA, while the paper does not specify licensing terms or detailed usage restrictions (Li et al., 8 Sep 2025).