---
title: ISRGC-Q Challenge Benchmark
url: https://www.emergentmind.com/topics/isrgc-q-challenge
type: topic
---

# ISRGC-Q Challenge Benchmark

Searching arXiv for the challenge paper and closely related terms.
The ISRGC-Q Challenge is a benchmark challenge on **Image Super-Resolution Generated Content Quality Assessment**, organized as part of the **VQualA 2025** workshop at ICCV 2025 and built on the **ISRGen-QA** dataset. Its central task is to predict the **perceptual quality score** of a super-resolved image as reflected by human **mean opinion score (MOS)**, with particular emphasis on outputs produced by modern **generative super-resolution** methods, including **GANs** and **diffusion models** [2509.06413].

## 1. Position within super-resolution image quality assessment

The challenge was introduced against the background that earlier SR-IQA databases, including **QADS, Waterloo, SISR-IQA, CVIU, RealSRQ,** and **SISAR**, are limited by **outdated SR models**, a **mismatch with current artifacts**, and **limited resolution and diversity**. In those earlier settings, the dominant distortions are typically **blur**, **over-smoothing**, **simple ringing**, and minor reconstruction errors. By contrast, modern generative SR systems produce artifact regimes that are qualitatively different: **hallucinated / false textures**, **over-sharpened edges**, **unnatural reconstruction patterns**, and **stochastic noise-like details from diffusion** [2509.06413].

This distinction is not merely taxonomic. It changes the target of quality assessment from conventional fidelity-sensitive distortion measurement to a more difficult estimation of **perceptual realism versus fidelity trade-offs**. A plausible implication is that an SR-IQA model trained on older restoration regimes may generalize poorly when confronted with visually plausible but semantically or structurally incorrect details.

The challenge therefore focuses on SR content generated by contemporary SR families rather than on legacy interpolation or early deep SR models. In the formulation used for the competition, participants were asked to estimate MOS directly from SR outputs, thereby treating perceptual quality prediction as the primary objective rather than as a proxy for PSNR-like fidelity.

## 2. ISRGen-QA dataset and subjective protocol

The underlying **ISRGen-QA** dataset contains **720** SR images at approximately **2K** resolution, specifically in the range
\[
2040 \times 1152 \sim 2040 \times 1440.
\]
It uses **19 HR reference images** from **DIV2K**, **76 LR reference images** produced by **bicubic** downsampling, and four scaling factors: \(\times 2\), \(\times 3\), \(\times 4\), and \(\times 8\). The dataset split is fixed as **576 images (80\%)** for training, **72 images (10\%)** for validation, and **72 images (10\%)** for testing [2509.06413].

The SR images were generated by **15 advanced SR algorithms** spanning five architectural families:

- **GAN-based (4)**: ESRGAN, Real-ESRGAN, BSRGAN, SeD
- **Diffusion-based (5)**: SR3, IDM, SRDiff, CDFormer, SAM-DiffSR
- **Transformer-based (4)**: SRNO, ATD-SR, SwinIR, CAMixerSR
- **Flow-based (1)**: BFSR
- **CNN-based (1)**: EDSR

This composition is important because each family contributes different artifact signatures. GAN-based methods are associated with **over-sharpened edges** and **hallucinated textures not present in the HR**; diffusion-based methods with **very rich, stochastic textures** and **noise-like high-frequency patterns**; transformer-based methods with **local over-smoothing in homogeneous regions** and **mild ringing / structural inconsistencies**; flow-based methods with **structured but sometimes misaligned details**; and CNN-based methods with more classical **over-smoothing** and **blur**.

Subjective annotation was performed by **23 human subjects (11 female, 12 male) from 5 countries**. After **anomaly filtering**, **21 participants’ scores** were retained as valid. For each image \(i\), MOS is defined as
\[
\text{MOS}(i) = \frac{1}{21} \sum_{k=1}^{21} s_{ik},
\]
where \(s_{ik}\) is the score assigned by subject \(k\) to image \(i\). The paper emphasizes subject diversity and anomaly filtering, but does not specify the exact viewing setup in the provided material.

## 3. Task formulation and evaluation protocol

The task is to predict, for each SR image, a **scalar perceptual quality score** approximating MOS. Participants were given the SR images and their corresponding **LR** and **HR** references. MOS labels were available only for the **training set**. No MOS values were released for the validation or test sets. The challenge is described as **primarily no-reference (NR) in spirit**, but the organizers released HR and LR images, and teams were free to use them during training or as auxiliary inputs. This clarifies a common misconception: the benchmark is not a strict blind-IQA competition in the narrow sense.

Performance was measured with two standard correlation criteria:

1. **Spearman Rank-Order Correlation Coefficient (SRCC)** for rank consistency
2. **Pearson Linear Correlation Coefficient (PLCC)** for linear agreement

The official leaderboard score is
\[
\text{Score} = 0.6 \times \text{SRCC} + 0.4 \times \text{PLCC}. \tag{1}
\]

The development phase used **train + validation images**, with MOS for the training set only. Participants could submit validation predictions to **Codalab** and observe **SRCC, PLCC, and Score**. This phase saw **193 submissions from 12 teams**. In the test phase, **72** test SR images were released without MOS. **5 teams** submitted predictions, and **4** teams that also provided **fact sheets & code** were admitted to the final ranking. Across the entire challenge, **108 participants registered** [2509.06413].

## 4. Submitted methods and design patterns

The four final methods span distinct SR-IQA design philosophies, ranging from compact NR architectures to high-resolution reference-aware systems.

| Team | Method profile | Official result |
|---|---|---|
| MICV | Hybrid **ViT + CNN**, SR-only, no-reference | **0.9638** (SRCC **0.9588**, PLCC **0.9714**) |
| ydy | Dual **ResNet-50** with **SR/LR**, residual maps, **BLIP-2**, MoE gating | **0.9429** (SRCC **0.9333**, PLCC **0.9572**) |
| QA-Veteran | **SigLIP2-base-patch16-NaFlex** VLM, blind SR-IQA, text-conditioned 5-class quality prediction | **0.9409** (SRCC **0.9277**, PLCC **0.9608**) |
| 2077 Agent | **ConvNeXt-Base + FPN + difference contrast network + Score Transformer**, SR/HR comparison | **0.9248** (SRCC **0.9096**, PLCC **0.9476**) |

**MICV**, the first-ranked team, used only the **SR image** as input. Its model combines **global self-attention** for long-range dependencies with **CNN modules** for local texture statistics. Training used randomized **\(448 \times 448\)** crops, **AdamW**, an initial learning rate of **\(1 \times 10^{-5}\)**, cosine annealing, and a **weighted sum (1:1) of PLCC loss and SRCC loss**. The model size is **6M parameters**, and training used **8× NVIDIA A100** for **200 epochs**.

**ydy** adopted a **reference-aware** strategy using both **SR and LR** images. Its architecture includes dual **ResNet50** encoders, residual guidance, **BLIP-2** semantic features, a **Semantic Refinement Module (SRM)**, a **Multiscale Semantic-guided Edge Residual Module (MSERM)**, and **Mixture-of-Experts (MoE) gating**. It predicts image quality by averaging over **30 patches** of size **\(128 \times 128\)**. The loss combines **L1 regression**, **Cross-Covariance Loss (CCL)**, and a **cosine similarity loss** to BLIP-2 features.

**QA-Veteran** treated the problem as **5-class quality classification** through a vision-language interface. Using **SigLIP2-base-patch16-NaFlex**, it encoded the image and matched it to prompts of the form *“a photo with {c quality}”* for the categories **bad**, **poor**, **fair**, **good**, and **perfect**. Class probabilities were converted into a scalar MOS estimate by expectation. Training used **AdamW**, a learning rate of **\(5 \times 10^{-6}\)**, cosine annealing, and a loss combining **fidelity loss**, **PLCC loss**, and **L1 loss**.

**2077 Agent** explicitly targeted **ultra-high-resolution** SR content. It first generated candidate regions on the HR image using **Selective Search**, then compared SR and HR features using **ROI Align**, a **difference contrast network**, and a **Score Transformer** with **Fourier and geometric positional encoding**. Rather than regress MOS directly, it predicted a **5-dimensional MOS frequency histogram** \((p_1,\dots,p_5)\) and computed
\[
\hat{q} = \sum_{c=1}^{5} p_c \times c.
\]
To supervise the histogram, the team constructed a **“minimal-variance” frequency histogram** consistent with the scalar MOS under the assumption of **21 raters** and minimized a **KL-divergence** objective [2509.06413].

## 5. Official results and technical interpretation

The official test-set results indicate uniformly high performance. All four final methods achieved **Score \(> 0.92\)**, with **PLCC between 0.9476 and 0.9714** and **SRCC between 0.9096 and 0.9588** [2509.06413].

Several technical conclusions follow directly from these numbers. First, the challenge appears difficult enough to differentiate architectures, but not so noisy that human perceptual structure is inaccessible to current models. Second, **PLCC > SRCC for all methods**. The paper interprets this as meaning that predictors are very strong in a **linear sense**, but still make errors in the **ranking of close-quality images**. This suggests that calibration to MOS is easier than perfect ordinal separation among visually similar SR outputs.

A further result concerns architectural efficiency. **MICV**, with only **6M parameters**, outperformed both **QA-Veteran** at **375.32M** parameters and **ydy** at **161M**. This indicates that in this benchmark, a compact design with explicit **global-attention plus local-texture** inductive bias can exceed much larger VLM-based or multi-branch reference-aware systems. The implication is methodological rather than absolute: model size alone is not a reliable predictor of SR-IQA performance when artifact structure is highly domain-specific.

The methods also reveal different ways of handling modern SR artifacts. **MICV** relies on joint global-local representation learning; **ydy** emphasizes **residual maps**, **edge distortions**, and semantic plausibility; **QA-Veteran** uses large-scale VLM priors and **resolution-adaptive patching**; and **2077 Agent** encodes the idea that some regions have **disproportionate impact** on perceived quality and should not be averaged uniformly.

## 6. Significance, limitations, and resources

The challenge’s primary significance lies in aligning SR-IQA evaluation with the **GAN/diffusion era** of super-resolution. Because ISRGen-QA is dominated by outputs from **state-of-the-art generative SR models**, success on the benchmark implies competence on artifact regimes that older SR-IQA datasets do not emphasize. This makes the challenge relevant not only for IQA model design but also for the development and selection of SR systems themselves.

At the same time, the benchmark has clear limitations. The dataset contains **720 images**, which is substantial for subjective SR-IQA but still modest relative to large-scale IQA corpora. The dataset also does not include explicit **artifact labels** such as “ringing” or “hallucination,” which limits fine-grained error analysis. Another limitation is that the paper does not report an ablation separating performance on **GAN** versus **diffusion** subsets, even though the dataset is explicitly designed to cover both families. These constraints suggest that future work may benefit from larger SR-IQA datasets, more content types, and explicit artifact-level annotation.

The benchmark also complicates the conventional boundary between blind and reference-based SR-IQA. Because HR and LR images were released, the final leaderboard mixes **SR-only**, **SR+LR**, and **SR+HR** usage patterns. This does not invalidate the ranking, but it means that the challenge measures performance under a broader design space than a strictly no-reference protocol would.

Reproducibility resources are publicly listed by the organizers. The **ISRGen-QA** project is available at:

`https://github.com/Lighting-YXLI/ISRGen-QA`

The **VQualA** portal is:

`https://vquala.github.io/`

The **Codalab** competition page is:

`https://codalab.lisn.upsaclay.fr/competitions/22924`

Taken together, the ISRGC-Q Challenge defines a specialized benchmark for **perceptual quality assessment of super-resolved images generated by modern generative SR methods**, provides a curated 2K-scale MOS-labeled dataset, and documents four strong but methodologically diverse solutions. Its results indicate that contemporary SR-IQA models can closely approximate human opinion on challenging generative SR content, while also showing that **rank-order consistency**, **artifact diversity**, and **protocol design** remain active research issues [2509.06413].

Source: https://www.emergentmind.com/topics/isrgc-q-challenge