VQualA 2025 Challenge Overview

• VQualA 2025 is a set of visual quality assessment competitions at ICCV 2025 that benchmark tasks like FIQA, ISRGC-Q, EVQA, and visual quality comparison for LMMs.
• The challenge introduces heterogeneous tracks that assess tasks ranging from no-reference MOS prediction to open-ended multimodal quality reasoning and engagement modeling.
• Evaluation protocols emphasize correlation with human judgments and tight efficiency constraints, bridging classical IQA and real-world quality assessments.

Searching arXiv for the VQualA 2025 challenge papers and related tracks. search_arxiv({"query":"VQualA 2025 Challenge ICCV 2025 Visual Quality Assessment workshop arXiv", "max_results": 10, "sort_by": "submittedDate"}) VQualA 2025 was a set of visual quality assessment competitions organized in conjunction with the ICCV 2025 Workshops. The released challenge reports show a program that ranged from Face Image Quality Assessment (FIQA) and Image Super-Resolution Generated Content Quality Assessment (ISRGC-Q) to Engagement Prediction for Short Videos (EVQA) and Visual Quality Comparison for Large Multimodal Models (LMMs). Taken together, these tracks treated visual quality not as a single scalar-regression problem, but as a family of tasks spanning Mean Opinion Score prediction, pairwise preference reasoning, multiple-choice comparison, and engagement modeling from real-world user interactions (Li et al., 8 Sep 2025, Ma et al., 25 Aug 2025, Li et al., 3 Sep 2025, Zhu et al., 11 Sep 2025).

1. Program scope and challenge taxonomy

One released report explicitly characterizes VQualA 2025 as a broader ecosystem of competitions, listing FIQA, EVQA-SnapUGC, Visual Quality Comparison for Large Multimodal Models, DIQA, and GenAI-Bench, with ISRGC-Q presented as one of the challenge tracks at the ICCV 2025 Workshops (Li et al., 8 Sep 2025). Within the subset of reports presently available, four tracks are described in technical detail: visual quality comparison for LMMs, image super-resolution generated content quality assessment, face image quality assessment, and engagement prediction for short videos.

These tracks differ substantially in target variable and supervision regime. FIQA is a no-reference perceptual quality prediction task for face images with arbitrary resolutions and realistic degradations. ISRGC-Q targets perceptual quality prediction for super-resolved images generated by contemporary super-resolution systems, especially GAN-based and diffusion-based methods. EVQA addresses short-video engagement prediction rather than classical subjective quality scoring. The visual quality comparison track asks LMMs to compare the perceptual quality of images across single-image, pairwise, and multi-image settings, using open-ended and detailed reasoning rather than ordinary recognition or captioning (Ma et al., 25 Aug 2025, Li et al., 8 Sep 2025, Li et al., 3 Sep 2025, Zhu et al., 11 Sep 2025).

Track	Core task	Reported participation
Visual Quality Comparison for LMMs	Visual quality comparison across single images, pairs, and multi-image groups using 2AFC-based binary preference and MCQs	Around 100 participants; five models demonstrated emerging capabilities
ISRGC-Q	Perceptual quality prediction for super-resolved images on ISRGen-QA	108 participants registered; 4 teams submitted valid solutions
FIQA	Lightweight MOS prediction on arbitrary-resolution face images	127 participants; 1519 final submissions
EVQA	Short-video engagement prediction on SnapUGC using ECR	97 participants; 15 valid test submissions

The resulting picture is of a benchmark family rather than a single homogeneous task. A plausible implication is that VQualA 2025 was designed to expose the mismatch between conventional IQA/VQA formulations and the broader operational settings in which quality-related judgments now arise.

2. Task formulations across the released tracks

The visual quality comparison track, summarized in "VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results" (Zhu et al., 11 Sep 2025), aimed to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. The benchmark comprises thousands of coarse-to-fine grained visual quality comparison tasks spanning single images, pairs, and multi-image groups. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions. The supplementary material further indicates that participant systems were explicitly designed for single-image quality assessment, pairwise comparison, and multi-image reasoning.

FIQA was framed as a no-reference perceptual quality prediction problem: given a face image, the goal is to predict its Mean Opinion Score on face images with arbitrary resolutions and realistic degradations, under strict efficiency limits of 0.5 GFLOPs and fewer than 5 million parameters (Ma et al., 25 Aug 2025). The challenge therefore combined perceptual fidelity with deployment-oriented constraints.

ISRGC-Q, presented in "VQualA 2025 Challenge on Image Super-Resolution Generated Content Quality Assessment: Methods and Results" (Li et al., 8 Sep 2025), targeted automatic perceptual quality prediction for super-resolved images. Its central motivation was that super-resolution is an ill-posed inverse problem, with a persistent tension between fidelity and naturalness, and that prior SR-IQA datasets do not adequately represent the artifacts introduced by modern generative priors. The track therefore emphasized SR images generated by GAN-based, diffusion-based, transformer-based, flow-based, and CNN-based methods.

EVQA, reported in "VQualA 2025 Challenge on Engagement Prediction for Short Videos: Methods and Results" (Li et al., 3 Sep 2025), departs most sharply from classical IQA/VQA. It was designed to predict short-video engagement for cold-start UGC content on platforms such as Snapchat Spotlight, using visual content, audio, and creator-provided metadata. The paper states explicitly that the challenge is not framed as a subjective visual quality assessment problem, but as a real engagement prediction task based on real user behavior.

3. Datasets, labels, and supervision regimes

The released VQualA 2025 tracks use markedly different data constructions and target labels. In the visual quality comparison challenge, the benchmark comprises thousands of coarse-to-fine grained tasks over single images, pairs, and multi-image groups, and each task requires models to provide accurate quality judgments. The supplementary description does not provide the full benchmark specification, the exact leaderboard, or the exact best-performing model and score, but it does make clear that the benchmark is intended to test both absolute quality judgment and relative preference reasoning (Zhu et al., 11 Sep 2025).

FIQA uses distinct training, validation, and test sets of 27,686, 1,000, and 889 images, respectively. Training and validation images were collected from CelebA and Flickr, while the test set was collected exclusively from Flickr. The short edge ranges from 224 to 1024 pixels, and the data include blur, noise, compression artifacts, and poor lighting. Importantly, the labels are proxy MOS values rather than a new subjective study: the organizers used DSL-FIQA to generate labels by extracting 20 random patches per image, averaging their predicted quality scores, and using the average as the image-level target (Ma et al., 25 Aug 2025).

ISRGC-Q is built on the ISRGen-QA database, which contains 720 super-resolved images at approximately $2040 \times 1152$ to $2040 \times 1440$, covering upscaling factors of $\times 2$, $\times 3$, $\times 4$, and $\times 8$. The reference side comprises 19 HR reference images and 76 LR reference images; HR references are selected from DIV2K and LR images are created using bicubic downsampling. The 720 SR images were generated by 15 advanced SR algorithms: 4 GAN-based, 5 diffusion-based, 4 transformer-based, 1 flow-based, and 1 CNN-based. Scores were collected from 23 human participants, of whom 21 remained after anomaly filtering, and the dataset was split into 576 training, 72 validation, and 72 test images (Li et al., 8 Sep 2025).

EVQA uses SnapUGC, a dataset of 120,651 short videos collected from publicly accessible short videos on Snapchat Spotlight, with 106,192 training, 6,000 validation, and 8,459 test videos. Durations range from 5 to 60 seconds, and only videos with more than 2,000 views were included. Each video comes with visual content, audio, and text metadata such as title and description. The challenge target is Engagement Continuation Rate, defined as

\text{ECR} = \mathbb{P}(\text{watch} > 5\text{s}),

and the paper reports that the correlation between ECR and NAWP is 0.928. For privacy, the competition ECR labels were derived by normalizing the ranking of the real ECR values (Li et al., 3 Sep 2025).

These heterogeneous supervision regimes are one of the defining features of VQualA 2025. Some tracks rely on human subjective ratings, some on proxy labels, some on pairwise or multiple-choice judgments, and some on population-scale behavioral signals.

4. Evaluation protocols and ranking criteria

The evaluation procedures vary by track and closely follow each task definition. In the visual quality comparison challenge, the official protocols emphasized 2AFC-based binary preference and MCQ evaluation. The accompanying summary indicates that this is consistent with a benchmark in which models must answer which image is better or choose from multiple candidates, but the excerpt does not provide the full benchmark equations, dataset size, or final leaderboard tables (Zhu et al., 11 Sep 2025).

FIQA used the average of SROCC and PLCC as the official ranking score,

\mathrm{Score} = (\mathrm{SROCC} + \mathrm{PLCC}) / 2,

with final awards determined by the testing-phase score. The test images were not released; instead, participants uploaded model definitions and weights, and the organizers ran the models on a hidden test server. This server-side protocol was motivated by fairness under the 0.5 GFLOPs and 5 million parameter constraints (Ma et al., 25 Aug 2025).

ISRGC-Q also used correlation-based evaluation, but with a different weighting:

\rm{Score} = 0.6\times \rm{SRCC} + 0.4\times \rm{PLCC}.

Participants were given 576 training images with references and 72 validation images without MOS in the development phase, then 72 test images with references but without MOS labels in the testing phase. During development, predictions were uploaded to CodaLab and participants received final score, SRCC, and PLCC (Li et al., 8 Sep 2025).

EVQA evaluates predictions against the ground-truth ECR ranking or score using SROCC and PLCC. A final score is reported in the results table, but the paper does not explicitly define the exact aggregation formula in the text. Unlike FIQA, EVQA imposed no model-size restrictions, because the organizers wanted to explore the upper bound of performance on the new multimodal task (Li et al., 3 Sep 2025).

A common theme across the released VQualA 2025 tracks is that evaluation favors correlation with human or user-centered targets over classical distortion minimization. Even when the labels differ—MOS, proxy MOS, pairwise preference, or real engagement behavior—the ranking criteria emphasize agreement with observed judgments rather than pixelwise reconstruction fidelity.

5. Competitive methods and reported results

The visual quality comparison challenge reports five participating teams after the organizers: ECNU-SJTU VQA Team, Digital Ocean, ActionLab, XiaomiMM, and Labubu. Their method titles indicate several recurring technical themes: fine-grained visual quality comparison via ensemble voting, joint optimization through a multi-modal LLM, two-stage fine-tuning for open-ended visual quality comparison, enhancing multi-image reasoning abilities via auxiliary visual scoring, and quality-token-based prompting. The challenge report states that around 100 participants submitted entries and that five models demonstrated the emerging capabilities of instruction-tuned LMMs on quality assessment, but the excerpt does not include numerical rankings or the exact best-performing model and score (Zhu et al., 11 Sep 2025).

ISRGC-Q reported four ranked teams. MICV placed first with an overall score of 0.9638, SRCC of 0.9588, and PLCC of 0.9714; ydy, QA-Veteran, and 2077 Agent followed. The winning MICV system used a Hybrid Vision Transformer + CNN for SR IQA and relied only on the SR image as input. The other top systems explored BLIP-2 assisted residual-guided quality assessment, blind SR IQA using a resolution-adaptive vision-LLM based on SigLIP2-NaFlex, and UltraR-IQA for ultra-high-resolution SR quality assessment (Li et al., 8 Sep 2025).

FIQA attracted 127 participants and 1519 final submissions. Thirteen final teams were reported, with ECNU-SJTU VQA Team ranked first at 0.9664, followed by MediaForensics at 0.9624 and Next at 0.9583. The baseline MobileNetV2 achieved 0.8309. The strongest approaches combined efficient backbones with staged training, self-training, knowledge distillation, prompt learning, progressive training, and local-global or spatial-frequency fusion, all while remaining within the 0.5 GFLOPs and 5 million parameter budget (Ma et al., 25 Aug 2025).

EVQA reported 97 participants and 15 valid test submissions. The baseline achieved a final score of 0.660, with SROCC 0.657 and PLCC 0.665. The best-performing systems were ECNU-SJTU VQA and IMCL-DAMO, treated as co-first place, with large multimodal model components such as Video-LLaMA2 and Qwen2.5-VL. A notable result is that HKUST-Cardiff-MI-BAAI was the best-performing non-LMM system, while a text-only system still achieved nontrivial performance, indicating that creator metadata carries substantial predictive signal (Li et al., 3 Sep 2025).

Across tracks, the published reports point to a recurring methodological pattern: specialized quality modeling increasingly depends on strong pretrained multimodal backbones, but strong results also come from carefully engineered lightweight systems when the challenge explicitly constrains efficiency.

6. Technical significance, misconceptions, and open problems

A common misconception would be to treat VQualA 2025 as a narrowly defined image-quality regression benchmark. The released track reports instead show a substantially broader research program. FIQA is a lightweight no-reference MOS predictor for in-the-wild faces; ISRGC-Q evaluates perceptual quality on SR outputs from modern generative methods; EVQA models user engagement rather than subjective MOS; and the LMM comparison track tests open-domain visual quality reasoning through preference-style and multiple-choice judgments (Ma et al., 25 Aug 2025, Li et al., 8 Sep 2025, Li et al., 3 Sep 2025, Zhu et al., 11 Sep 2025).

The challenge family also highlights several unresolved technical issues. In the LMM comparison track, the methodological interpretation provided in the summary suggests ongoing difficulty with fine-grained perceptual discrimination, robust multi-image comparison, open-domain visual quality reasoning, and reliable calibration. In FIQA, the use of DSL-FIQA-generated proxy labels rather than a new subjective study suggests a limitation of current data collection pipelines. ISRGC-Q implies that existing SR-IQA datasets are outdated relative to GAN-based and diffusion-based SR, and that rank consistency remains harder than linear fit. EVQA shows that MOS-style VQA labels are not a satisfactory proxy for real short-video popularity and that multimodal cold-start engagement prediction remains strongly dependent on visual, audio, and textual cues (Zhu et al., 11 Sep 2025, Ma et al., 25 Aug 2025, Li et al., 8 Sep 2025, Li et al., 3 Sep 2025).

More broadly, VQualA 2025 documents a shift in the notion of “quality assessment.” Quality is no longer limited to scalar fidelity estimation under synthetic distortions. It now includes efficiency-constrained perceptual prediction, quality-aware reasoning by LMMs, human judgment of artifacts introduced by generative SR systems, and user-behavior-driven engagement modeling. This suggests that future visual quality benchmarks will continue to move toward heterogeneous supervision, multimodal inputs, and evaluation targets that are closer to real human decisions than to classical full-reference distortion metrics.