ISRGC-Q Challenge Benchmark

• ISRGC-Q Challenge is a benchmark assessing perceptual quality of super-resolved images, focusing on modern generative methods with MOS estimation.
• It leverages the ISRGen-QA dataset of 720 images generated from 15 advanced SR algorithms including GANs, diffusion, transformers, flow, and CNN-based models.
• Evaluation uses Spearman and Pearson correlation metrics to analyze quality prediction performance and address challenges in artifact diversity.

Searching arXiv for the challenge paper and closely related terms. The ISRGC-Q Challenge is a benchmark challenge on Image Super-Resolution Generated Content Quality Assessment, organized as part of the VQualA 2025 workshop at ICCV 2025 and built on the ISRGen-QA dataset. Its central task is to predict the perceptual quality score of a super-resolved image as reflected by human mean opinion score (MOS), with particular emphasis on outputs produced by modern generative super-resolution methods, including GANs and diffusion models (Li et al., 8 Sep 2025).

1. Position within super-resolution image quality assessment

The challenge was introduced against the background that earlier SR-IQA databases, including QADS, Waterloo, SISR-IQA, CVIU, RealSRQ, and SISAR, are limited by outdated SR models, a mismatch with current artifacts, and limited resolution and diversity. In those earlier settings, the dominant distortions are typically blur, over-smoothing, simple ringing, and minor reconstruction errors. By contrast, modern generative SR systems produce artifact regimes that are qualitatively different: hallucinated / false textures, over-sharpened edges, unnatural reconstruction patterns, and stochastic noise-like details from diffusion (Li et al., 8 Sep 2025).

This distinction is not merely taxonomic. It changes the target of quality assessment from conventional fidelity-sensitive distortion measurement to a more difficult estimation of perceptual realism versus fidelity trade-offs. A plausible implication is that an SR-IQA model trained on older restoration regimes may generalize poorly when confronted with visually plausible but semantically or structurally incorrect details.

The challenge therefore focuses on SR content generated by contemporary SR families rather than on legacy interpolation or early deep SR models. In the formulation used for the competition, participants were asked to estimate MOS directly from SR outputs, thereby treating perceptual quality prediction as the primary objective rather than as a proxy for PSNR-like fidelity.

2. ISRGen-QA dataset and subjective protocol

The underlying ISRGen-QA dataset contains 720 SR images at approximately 2K resolution, specifically in the range

$2040 \times 1152 \sim 2040 \times 1440.$

It uses 19 HR reference images from DIV2K, 76 LR reference images produced by bicubic downsampling, and four scaling factors: $\times 2$, $\times 3$, $\times 4$, and $\times 8$. The dataset split is fixed as 576 images (80\%) for training, 72 images (10\%) for validation, and 72 images (10\%) for testing (Li et al., 8 Sep 2025).

The SR images were generated by 15 advanced SR algorithms spanning five architectural families:

• GAN-based (4): ESRGAN, Real-ESRGAN, BSRGAN, SeD
• Diffusion-based (5): SR3, IDM, SRDiff, CDFormer, SAM-DiffSR
• Transformer-based (4): SRNO, ATD-SR, SwinIR, CAMixerSR
• Flow-based (1): BFSR
• CNN-based (1): EDSR

This composition is important because each family contributes different artifact signatures. GAN-based methods are associated with over-sharpened edges and hallucinated textures not present in the HR; diffusion-based methods with very rich, stochastic textures and noise-like high-frequency patterns; transformer-based methods with local over-smoothing in homogeneous regions and mild ringing / structural inconsistencies; flow-based methods with structured but sometimes misaligned details; and CNN-based methods with more classical over-smoothing and blur.

Subjective annotation was performed by 23 human subjects (11 female, 12 male) from 5 countries. After anomaly filtering, 21 participants’ scores were retained as valid. For each image $i$, MOS is defined as

$$\text{MOS}(i) = \frac{1}{21} \sum_{k=1}^{21} s_{ik},$$

where $s_{ik}$ is the score assigned by subject $k$ to image $i$. The paper emphasizes subject diversity and anomaly filtering, but does not specify the exact viewing setup in the provided material.

3. Task formulation and evaluation protocol

The task is to predict, for each SR image, a scalar perceptual quality score approximating MOS. Participants were given the SR images and their corresponding LR and HR references. MOS labels were available only for the training set. No MOS values were released for the validation or test sets. The challenge is described as primarily no-reference (NR) in spirit, but the organizers released HR and LR images, and teams were free to use them during training or as auxiliary inputs. This clarifies a common misconception: the benchmark is not a strict blind-IQA competition in the narrow sense.

Performance was measured with two standard correlation criteria:

1. Spearman Rank-Order Correlation Coefficient (SRCC) for rank consistency
2. Pearson Linear Correlation Coefficient (PLCC) for linear agreement

The official leaderboard score is

$\times 2$0

The development phase used train + validation images, with MOS for the training set only. Participants could submit validation predictions to Codalab and observe SRCC, PLCC, and Score. This phase saw 193 submissions from 12 teams. In the test phase, 72 test SR images were released without MOS. 5 teams submitted predictions, and 4 teams that also provided fact sheets & code were admitted to the final ranking. Across the entire challenge, 108 participants registered (Li et al., 8 Sep 2025).

4. Submitted methods and design patterns

The four final methods span distinct SR-IQA design philosophies, ranging from compact NR architectures to high-resolution reference-aware systems.

Team	Method profile	Official result
MICV	Hybrid ViT + CNN, SR-only, no-reference	0.9638 (SRCC 0.9588, PLCC 0.9714)
ydy	Dual ResNet-50 with SR/LR, residual maps, BLIP-2, MoE gating	0.9429 (SRCC 0.9333, PLCC 0.9572)
QA-Veteran	SigLIP2-base-patch16-NaFlex VLM, blind SR-IQA, text-conditioned 5-class quality prediction	0.9409 (SRCC 0.9277, PLCC 0.9608)
2077 Agent	ConvNeXt-Base + FPN + difference contrast network + Score Transformer, SR/HR comparison	0.9248 (SRCC 0.9096, PLCC 0.9476)

MICV, the first-ranked team, used only the SR image as input. Its model combines global self-attention for long-range dependencies with CNN modules for local texture statistics. Training used randomized $\times 2$1 crops, AdamW, an initial learning rate of $\times 2$2, cosine annealing, and a weighted sum (1:1) of PLCC loss and SRCC loss. The model size is 6M parameters, and training used 8× NVIDIA A100 for 200 epochs.

ydy adopted a reference-aware strategy using both SR and LR images. Its architecture includes dual ResNet50 encoders, residual guidance, BLIP-2 semantic features, a Semantic Refinement Module (SRM), a Multiscale Semantic-guided Edge Residual Module (MSERM), and Mixture-of-Experts (MoE) gating. It predicts image quality by averaging over 30 patches of size $\times 2$3. The loss combines L1 regression, Cross-Covariance Loss (CCL), and a cosine similarity loss to BLIP-2 features.

QA-Veteran treated the problem as 5-class quality classification through a vision-language interface. Using SigLIP2-base-patch16-NaFlex, it encoded the image and matched it to prompts of the form “a photo with {c quality}” for the categories bad, poor, fair, good, and perfect. Class probabilities were converted into a scalar MOS estimate by expectation. Training used AdamW, a learning rate of $\times 2$4, cosine annealing, and a loss combining fidelity loss, PLCC loss, and L1 loss.

2077 Agent explicitly targeted ultra-high-resolution SR content. It first generated candidate regions on the HR image using Selective Search, then compared SR and HR features using ROI Align, a difference contrast network, and a Score Transformer with Fourier and geometric positional encoding. Rather than regress MOS directly, it predicted a 5-dimensional MOS frequency histogram $\times 2$5 and computed

$\times 2$6

To supervise the histogram, the team constructed a “minimal-variance” frequency histogram consistent with the scalar MOS under the assumption of 21 raters and minimized a KL-divergence objective (Li et al., 8 Sep 2025).

5. Official results and technical interpretation

The official test-set results indicate uniformly high performance. All four final methods achieved Score $\times 2$7, with PLCC between 0.9476 and 0.9714 and SRCC between 0.9096 and 0.9588 (Li et al., 8 Sep 2025).

Several technical conclusions follow directly from these numbers. First, the challenge appears difficult enough to differentiate architectures, but not so noisy that human perceptual structure is inaccessible to current models. Second, PLCC > SRCC for all methods. The paper interprets this as meaning that predictors are very strong in a linear sense, but still make errors in the ranking of close-quality images. This suggests that calibration to MOS is easier than perfect ordinal separation among visually similar SR outputs.

A further result concerns architectural efficiency. MICV, with only 6M parameters, outperformed both QA-Veteran at 375.32M parameters and ydy at 161M. This indicates that in this benchmark, a compact design with explicit global-attention plus local-texture inductive bias can exceed much larger VLM-based or multi-branch reference-aware systems. The implication is methodological rather than absolute: model size alone is not a reliable predictor of SR-IQA performance when artifact structure is highly domain-specific.

The methods also reveal different ways of handling modern SR artifacts. MICV relies on joint global-local representation learning; ydy emphasizes residual maps, edge distortions, and semantic plausibility; QA-Veteran uses large-scale VLM priors and resolution-adaptive patching; and 2077 Agent encodes the idea that some regions have disproportionate impact on perceived quality and should not be averaged uniformly.

6. Significance, limitations, and resources

The challenge’s primary significance lies in aligning SR-IQA evaluation with the GAN/diffusion era of super-resolution. Because ISRGen-QA is dominated by outputs from state-of-the-art generative SR models, success on the benchmark implies competence on artifact regimes that older SR-IQA datasets do not emphasize. This makes the challenge relevant not only for IQA model design but also for the development and selection of SR systems themselves.

At the same time, the benchmark has clear limitations. The dataset contains 720 images, which is substantial for subjective SR-IQA but still modest relative to large-scale IQA corpora. The dataset also does not include explicit artifact labels such as “ringing” or “hallucination,” which limits fine-grained error analysis. Another limitation is that the paper does not report an ablation separating performance on GAN versus diffusion subsets, even though the dataset is explicitly designed to cover both families. These constraints suggest that future work may benefit from larger SR-IQA datasets, more content types, and explicit artifact-level annotation.

The benchmark also complicates the conventional boundary between blind and reference-based SR-IQA. Because HR and LR images were released, the final leaderboard mixes SR-only, SR+LR, and SR+HR usage patterns. This does not invalidate the ranking, but it means that the challenge measures performance under a broader design space than a strictly no-reference protocol would.

Reproducibility resources are publicly listed by the organizers. The ISRGen-QA project is available at:

https://github.com/Lighting-YXLI/ISRGen-QA

The VQualA portal is:

https://vquala.github.io/

The Codalab competition page is:

https://codalab.lisn.upsaclay.fr/competitions/22924

Taken together, the ISRGC-Q Challenge defines a specialized benchmark for perceptual quality assessment of super-resolved images generated by modern generative SR methods, provides a curated 2K-scale MOS-labeled dataset, and documents four strong but methodologically diverse solutions. Its results indicate that contemporary SR-IQA models can closely approximate human opinion on challenging generative SR content, while also showing that rank-order consistency, artifact diversity, and protocol design remain active research issues (Li et al., 8 Sep 2025).