---
title: 'ISRGen-QA: Modern SR-IQA Benchmark'
url: https://www.emergentmind.com/topics/isrgen-qa-dataset
type: topic
---

# ISRGen-QA: Modern SR-IQA Benchmark

ISRGen-QA, short for **Image Super-Resolution Generated Content Quality Assessment**, is a super-resolution image quality assessment dataset introduced as the foundation of the **VQualA 2025 Challenge on Image Super-Resolution Generated Content Quality Assessment (ISRGC-Q)**. It is designed to evaluate perceptual quality on super-resolved images produced by contemporary generative super-resolution pipelines, especially **GAN-based** and **diffusion-based** methods, rather than the older interpolation- or early CNN-centered outputs that dominate earlier SR-IQA resources. The dataset contains **720 super-resolved images** at approximately **2K resolution**, derived from **19 high-resolution reference images** and **76 low-resolution reference images**, with human perceptual judgments aggregated as **MOS** and used as the target for quality prediction [2509.06413].

## 1. Motivation and positioning within SR-IQA

ISRGen-QA was created in response to a specific shift in the super-resolution literature: modern SR systems increasingly synthesize texture and structure through strong generative priors, so perceptual quality can no longer be treated primarily as a problem of classical fidelity to a reference. The motivating paper frames SR as an **ill-posed inverse problem** in which multiple plausible high-resolution reconstructions may correspond to a single low-resolution input. In that setting, perceptual failure modes include not only blur, noise, and compression, but also **over-sharpened edges, hallucinated or false textures, unnatural reconstruction patterns, texture inconsistencies, and edge distortions**, together with the broader **fidelity–naturalness** tradeoff [2509.06413].

The dataset is explicitly positioned against earlier SR-IQA collections such as **QADS, Waterloo, SISR-IQA, CVIU, RealSRQ, and SISAR**. Those resources are described as being dominated by interpolation methods, early learning-based approaches, or older CNN SR models, with only limited representation of modern generative SR. The paper notes that in prior datasets the newest included methods are around **2019–2020**, and that **SRGAN is essentially the only GAN-based representative**. ISRGen-QA therefore functions as a benchmark update: it aims to make SR-IQA reflect the artifact regimes of current generative SR rather than legacy restoration pipelines.

## 2. Construction and composition

ISRGen-QA contains **720 super-resolved images** at image sizes in the range **\(2040 \times 1152 \sim 2040 \times 1440\)**. The source material consists of **19 high-resolution reference images** selected from **DIV2K**. These are converted into **76 low-resolution reference images** by **bicubic** downsampling at four scales: **\(\times 2\), \(\times 3\), \(\times 4\), and \(\times 8\)**. The low-resolution references are then processed by **15 advanced SR algorithms**, producing the final SR image set [2509.06413].

The included SR methods span several architectural families:

| Family | Methods |
|---|---|
| GAN-based | ESRGAN, Real-ESRGAN, BSRGAN, SeD |
| Diffusion-based | SR3, IDM, SRDiff, CDFormer, SAM-DiffSR |
| Transformer-based | SRNO, ATD-SR, SwinIR, CAMixerSR |
| Flow-based | BFSR |
| CNN-based | EDSR |

This construction deliberately mixes **GAN-based**, **diffusion-based**, **transformer-based**, **flow-based**, and **CNN-based** systems. The intent is not only architectural breadth, but also distortion diversity. The paper states that the reported total of **720** SR outputs does not come with a finer published breakdown by algorithm or scale; similarly, it does not specify how many samples each method contributes beyond the total count and the fact that all four scales are covered. This suggests that the benchmark is primarily defined by aggregate coverage rather than by a fully enumerated factorial design.

The dataset is partitioned using an **80/10/10** split into **576 training images**, **72 validation images**, and **72 test images**. The paper presents this split as the standard experimental basis for reproducible model development and challenge evaluation [2509.06413].

## 3. Subjective annotation and target labels

The target labels in ISRGen-QA are **MOS** values derived from human opinion scores. Subjective annotation involved **23 human participants**, consisting of **11 female and 12 male** raters from **5 different countries** and of various ages. After **anomaly filtering**, the final valid scores were taken from **21 participants**, and the challenge paper consistently treats the resulting MOS as the prediction target [2509.06413].

The annotation protocol, however, is only partially documented. The paper does **not** specify the exact viewing setup, display conditions, presentation method, rating scale labels, whether the design was single-stimulus or double-stimulus, or the anomaly filtering criterion. It also does **not** define MOS with an explicit formula in the dataset description, and **DMOS** is not used. Likewise, the paper does not report label-distribution histograms, inter-rater agreement coefficients, confidence intervals, or score variances. As a result, ISRGen-QA is well specified as a benchmark corpus with human perceptual targets, but less fully specified as a psychophysical dataset than some dedicated IQA studies.

A common misconception is that ISRGen-QA is primarily a no-reference dataset because several strong submissions emphasize blind assessment. The paper does not support that characterization. During challenge development and testing, participants receive SR images together with their corresponding **HR and LR reference images**, and the benchmark is therefore best described as **open with respect to reference usage** rather than restricted to a single FR, RR, or NR regime.

## 4. Challenge protocol and evaluation framework

ISRGen-QA is operationalized through the **ISRGC-Q** task in the **VQualA 2025 Challenge**, which asks participants to predict perceptual quality scores for SR images so that predictions align with human MOS. The official evaluation metrics are **Spearman rank-order correlation coefficient (SRCC)** and **Pearson linear correlation coefficient (PLCC)**. Final ranking uses the weighted score

\[
\mathrm{Score} = 0.6 \times \mathrm{SRCC} + 0.4 \times \mathrm{PLCC}.
\]

An ideal method is therefore one that is simultaneously strong in monotonic ranking fidelity and linear score regression [2509.06413].

The challenge had two phases. In the **development phase**, participants received the **576 training SR images** with MOS labels, along with the corresponding **HR and LR references**, and also received **72 validation SR images** without MOS labels. In the **testing phase**, they received **72 test SR images** and their HR and LR references, and final ranking was computed using the weighted score above. The challenge registered **108 participants** overall, received **193 submissions from 12 teams** during development, and obtained **5** final test submissions, of which **4** were retained in the official ranking because only those four teams submitted the required **fact sheets and source codes**.

The benchmark is methodologically notable because it accommodates different input regimes. The winning **MICV** method used **only the SR image**; **QA-Veteran** explicitly pursued a **blind/no-reference** approach; **ydy** used both **SR and LR**; and **2077 Agent** used **SR and HR**. This makes ISRGen-QA a benchmark for perceptual prediction under heterogeneous reference assumptions rather than a benchmark tied to one canonical IQA setting.

## 5. Empirical behavior and challenge results

The official test-set leaderboard shows uniformly high performance but also persistent difficulty in fine-grained ranking. The final results were: **MICV** with overall **0.9638**, **ydy** with **0.9429**, **QA-Veteran** with **0.9409**, and **2077 Agent** with **0.9248**. Their corresponding **PLCC** values ranged from **0.9476** to **0.9714**, while **SRCC** ranged from **0.9096** to **0.9588** [2509.06413].

A central empirical observation in the paper is that **PLCC is consistently higher than SRCC** for the top methods. The authors interpret this as evidence that score regression is easier than preserving perfect perceptual rank ordering across the full quality range, especially when images have similar perceptual quality. This makes ISRGen-QA non-trivial despite the high absolute scores: top systems can learn the global MOS mapping well, yet still struggle with local ordering consistency.

The methods themselves also indicate the range of modeling priors that ISRGen-QA can support. **MICV** uses a **hybrid Vision Transformer + CNN** and only the SR image; **ydy** combines **ResNet50** backbones, **BLIP-2 semantic features**, SR-LR residual guidance, and a **Mixture-of-Experts** mechanism; **QA-Veteran** formulates blind SR-IQA as a **resolution-adaptive vision-language** problem using **SigLIP2-NaFlex**; and **2077 Agent** emphasizes **ultra-high-resolution** perception with candidate-region sampling, **ConvNeXt-Base**, **FPN**, and histogram prediction over discrete quality levels. This variety suggests that ISRGen-QA is compatible with global-context models, blind IQA formulations, residual-reference methods, and region-aware perceptual aggregation.

## 6. Significance, limitations, and scope

ISRGen-QA is significant because it updates SR-IQA benchmarking to match the perceptual behavior of **generated super-resolution content**, especially outputs from **GANs** and **diffusion models**. Its design foregrounds artifact classes that are weakly represented in older SR-IQA datasets, including **hallucinated textures**, **false detail**, **unnatural structures**, and **edge anomalies**, and thereby reorients quality assessment toward the perceptual regimes of current SR research [2509.06413].

At the same time, the challenge paper leaves several dataset-analysis questions open. It does not provide a detailed subjective-study protocol, label-distribution analysis, inter-rater agreement, per-distortion breakdowns, per-model-family breakdowns, or per-scale breakdowns. It also does not present baseline results from prior IQA literature on the dataset itself, so the main comparative evidence comes from the four final challenge teams. Nor does it provide a dedicated empirical artifact-by-artifact comparison between GAN-generated and diffusion-generated SR outputs; that distinction appears primarily as motivation rather than as a reported analysis result.

The public-facing role of ISRGen-QA is therefore twofold. First, it is a modern SR-IQA dataset built around **contemporary generative SR** rather than legacy SR outputs. Second, it is a challenge benchmark that exposes an open methodological question: whether perceptual quality prediction for generated SR is best approached by blind IQA, reference-assisted quality assessment, multimodal semantic modeling, or region-aware ultra-high-resolution analysis. The project is reported as publicly available through the ISRGen-QA repository at `https://github.com/Lighting-YXLI/ISRGen-QA`, while the paper does not specify licensing terms or detailed usage restrictions [2509.06413].

Source: https://www.emergentmind.com/topics/isrgen-qa-dataset