AgenticIQA-Eval Benchmark

• AgenticIQA-Eval is the first benchmark that evaluates VLM-based IQA agents by testing planning, execution, and summarization via multiple-choice questions.
• It decomposes image quality assessment into subtasks such as distortion detection, analysis, tool selection, and execution to measure agentic perception.
• The benchmark uses 1000 expert-verified MCQs to diagnose reasoning capabilities, highlighting challenges like distortion reasoning bottlenecks.

AgenticIQA-Eval is the benchmark introduced alongside the AgenticIQA framework for evaluating VLM-based IQA agents at the level of internal agent behavior rather than only final scalar quality prediction. It is described as “the first benchmark for assessing the planning, execution, and summarization capabilities of VLM-based IQA agents” and as a benchmark “measuring planning accuracy, execution precision, and summarization reliability through multiple-choice questions (MCQs)” (Zhu et al., 30 Sep 2025). Its central purpose is to diagnose whether a model can act as a modular, query-aware image-quality assessor that interprets the query, reasons about distortions, selects appropriate IQA tools, and synthesizes intermediate perceptual evidence into a final decision.

1. Concept and motivation

AgenticIQA-Eval arises from a mismatch between conventional IQA evaluation and the requirements of an agentic IQA system. Traditional FR-IQA and NR-IQA evaluation emphasizes scalar alignment with MOS through measures such as SRCC and PLCC, while some VLM-oriented settings emphasize low-level visual QA or description generation. The benchmark’s motivating claim is that these settings do not evaluate whether a system can perform the sequence of decisions required by a modular IQA pipeline: understand the query, determine whether the task is IQA or something else, infer or extract distortions, choose appropriate tools, execute them when needed, and integrate heterogeneous evidence into an answer (Zhu et al., 30 Sep 2025).

The benchmark therefore targets adaptive, modular, query-aware IQA rather than fixed black-box scoring. The paper explicitly argues that score-only metrics ignore reasoning structure, that interpretation and scoring are often disconnected, that static benchmarks do not test adaptivity, and that low-level VLM benchmarks such as Q-Bench and LLVisionQA still do not directly test planner/executor/summarizer decomposition or tool selection. This suggests a shift from evaluating IQA systems as scalar predictors to evaluating them as agentic perception systems (Zhu et al., 30 Sep 2025).

2. Relation to the AgenticIQA architecture

AgenticIQA itself follows a plan–execute–summarize design with three VLM-driven modules—Planner, Executor, and Summarizer—and decomposes IQA into four subtasks: distortion detection, distortion analysis, tool selection, and tool execution. The benchmark mirrors this architecture directly, so its tracks are not auxiliary tasks detached from the method but probes of the framework’s core decision points (Zhu et al., 30 Sep 2025).

Formally, the executor subtasks are written as

$$M_i = \mathcal{E}_i(x, t, P_t(i), \mathcal{T}), \quad i \in \{dd, da, ts, te\},$$

with distortion detection

$D = \mathcal{E}_{dd}(x, t_{dd}),$

distortion analysis

$$A_i = \left\{ \left(d_i, l_i, r_i\right) \mid d_i \in D_k \right\} = \mathcal{E}_{da}(x, t_{da}, D_k, O_k),$$

tool selection

$T_i = \mathcal{E}_{ts}(d_i, t_{ts}, \mathcal{T}),$

and tool execution

$\hat{q}_i = \mathcal{E}_{te}(x, T_i).$

Here $$l_i \in \{\text{none, slight, moderate, severe, extreme}\}$$ is the severity level and $r_i$ is a textual explanation (Zhu et al., 30 Sep 2025).

The Planner determines fields such as query type (IQA vs Other), reference mode (Full-Reference or No-Reference), query scope (Global or object-specific), distortion source (Explicit or Inferred), and boolean activation flags for distortion detection, distortion analysis, tool selection, and tool execution. The surrounding framework specifies JSON outputs for planner, distortion detection, distortion analysis, tool selection, and summarizer modules, but AgenticIQA-Eval itself scores capability through MCQs rather than direct structured-output matching (Zhu et al., 30 Sep 2025).

The Executor operates over a tool library $\mathcal{T}$ drawn from IQA-PyTorch. Listed FR tools include TOPIQ_FR, AHIQ, FSIM, LPIPS, DISTS, WaDIQaM_FR, PieAPP, MS-SSIM, GMSD, SSIM, CKDN, VIF, PSNR, and VSI. Listed NR tools include QAlign, CLIPIQA, HyperIQA, TReS, MUSIQ, NIMA, BRISQUE, NIQE, MANIQA, and LIQE; the paper notes inconsistent spelling for UNIQIE / UNIQUE. The appendix further states that tools have different distortion strengths, for example TOPIQ across blur, color, compression, noise, brightness, spatial distortions, and sharpness/contrast, QAlign across blur, color distortions, noise, brightness, spatial distortions, and sharpness, UNIQUE on lens blur, JPEG/JPEG2000, denoise artifact, spatial distortions, and contrast, LIQE on color diffusion, and DISTS on Gaussian blur (Zhu et al., 30 Sep 2025).

3. Benchmark construction and annotation

AgenticIQA-Eval contains 1000 curated MCQ samples. Its image pool combines 500 distorted images from MICBench, which contains “content from diverse sources with authentic and generative degradations,” with 500 pristine images from the Waterloo Exploration Database that are synthetically degraded with one or two random distortions following the protocol of Descriptive IQA in the Wild. The final composition is 750 NR-IQA instances and 250 FR-IQA instances (Zhu et al., 30 Sep 2025).

The benchmark is organized into three tracks aligned with the planner–executor–summarizer decomposition.

Track	Samples	Evaluated capability
Planner	250	Subtask configurations and valid evaluation plans
Executor	500	250 distortion identification/severity + 250 tool selection
Summarizer	250	Use of intermediate perceptual state $M_t$ to produce accurate responses

Each sample is an MCQ with a single ground-truth answer. The question formats are What, How, Which, and Yes/No, and the appendix states that the benchmark maintains a balanced distribution across these question types. The appendix also states coverage across FR and NR scenarios, distortion types such as blur, noise, compression, and planning configurations (Zhu et al., 30 Sep 2025).

Curation is manual and expert-verified. The paper states that all MCQs are manually curated, every question is cross-verified by at least two expert annotators, and the appendix further specifies that each question is independently verified by a second annotator with disagreements resolved by a third expert adjudicator. The answer options are designed to be “plausible yet discriminative,” so the benchmark is intended to measure reasoning rather than shortcut pattern matching (Zhu et al., 30 Sep 2025).

Representative examples illustrate the three tracks. Planner questions test query interpretation, including object extraction from prompts such as “What is the primary distortion affecting the boat clarity?” Executor questions test distortion severity or tool appropriateness, for example whether NIMA is suitable for sharpness assessment. Summarizer questions require inference from distortion analysis and tool scores, such as attributing object obscurity to blur or out-of-focus artifacts (Zhu et al., 30 Sep 2025).

4. Evaluation protocol, interfaces, and measured abilities

The benchmark’s primary metric is accuracy, reported independently for each subtrack and then overall. The paper is explicit that AgenticIQA-Eval does not use LLM-as-a-judge, free-form rubric scoring, SRCC/PLCC/KRCC, or explanation-similarity metrics for the benchmark proper. Labels are human-authored and human-verified, while scoring is automatic exact-answer matching over MCQs (Zhu et al., 30 Sep 2025).

This design makes the benchmark narrower and more controlled than fully open-ended agent execution. Planning-oriented tasks are present; tool-use and tool-selection reasoning are present; instruction-following is present insofar as the model must interpret the query and role-specific behavior. By contrast, open-ended explanation generation is not the primary scoring format, multi-turn tasks are not described, and trajectory-level evaluation is not explicitly described. The benchmark is therefore best characterized as a single-turn, MCQ-based, component-level evaluation benchmark for planning, execution-related reasoning, and summarization over evidence (Zhu et al., 30 Sep 2025).

The benchmark is nevertheless tightly coupled to the framework’s structured interfaces. The Planner is expected to emit JSON fields such as query_type, query_scope, distortion_source, distortions, reference_mode, required_tool, and a plan subobject containing boolean switches for distortion_detection, distortion_analysis, tool_selection, and tool_execute. Distortion detection outputs a distortion_set; distortion analysis outputs typed distortions, severity labels, and explanations; tool selection outputs a selected_tools mapping; and the Summarizer outputs either a multiple-choice final_answer with quality_reasoning or a free-form reasoning-plus-answer pair for instruction data (Zhu et al., 30 Sep 2025).

The surrounding AgenticIQA framework also specifies normalization and fusion machinery for tool outputs. Tool scores are mapped to a common $[1,5]$ scale through a five-parameter monotonic logistic transformation,

$D = \mathcal{E}_{dd}(x, t_{dd}),$0

where $D = \mathcal{E}_{dd}(x, t_{dd}),$1 is the raw tool score. The paper further defines a summarizer-side fusion using

$D = \mathcal{E}_{dd}(x, t_{dd}),$2

and perceptual weights

$D = \mathcal{E}_{dd}(x, t_{dd}),$3

with $D = \mathcal{E}_{dd}(x, t_{dd}),$4 in experiments, but notes that the printed final score equation contains a typesetting error. These formulas belong to the surrounding framework rather than the benchmark metric itself (Zhu et al., 30 Sep 2025).

5. Empirical results

The main benchmark results are reported as average accuracy of agent-level performance under the AgenticIQA framework. GPT-4o is the strongest model overall at 76.30%, while Qwen2.5-VL* is the strongest open-source model at 75.40%. Human performance is 81.80%, leaving a visible human–model gap. Other reported systems include mPLUG-Owl3, InternVL2.5, LLaVA-OneVision, Qwen2.5-VL, Q-Instruct, Q-SiT, Claude-3.5-Sonnet, and Gemini-2.0-Flash (Zhu et al., 30 Sep 2025).

The subtrack pattern is more informative than the overall ranking. Planning and summarization are generally easier than distortion and tool subtasks for many VLMs. The hardest component is distortion reasoning: the best Executor-Distortion score is 64.40% from GPT-4o. Tool selection can remain relatively strong even when distortion reasoning is weaker; for example, LLaVA-OneVision reaches 77.20% on Executor-Tool despite a substantially lower distortion score. The paper treats this as evidence that low-level perceptual reasoning remains a bottleneck for agentic VLMs (Zhu et al., 30 Sep 2025).

The effect of agentic instruction tuning is also localized rather than uniform. The paper compares base Qwen2.5-VL with Qwen2.5-VL* and reports an overall improvement from 73.20% to 75.40%, with the largest gain in Executor Distortion: 55.60% $D = \mathcal{E}_{dd}(x, t_{dd}),$5 63.20%, a +7.6 increase. Planner improves from 74.40% to 76.80%, Summarizer from 84.80% to 85.20%, but Executor-Tool decreases from 78.00% to 76.40%. The paper therefore concludes that the major benefit of agentic tuning is in distortion reasoning, not tool selection (Zhu et al., 30 Sep 2025).

One model, Q-SiT, degrades sharply on this benchmark, reaching 29.60% overall, which the paper attributes to fine-tuning on rigid fixed-format instructions that harms flexible agentic reasoning. By contrast, general-purpose VLMs show moderate planning and summarization ability due to stronger language competence, even when their low-level perceptual decisions remain weaker (Zhu et al., 30 Sep 2025).

6. Interpretive significance, neighboring evaluation paradigms, and limitations

AgenticIQA-Eval is significant because it redefines IQA benchmarking around behavioral decomposition. Rather than asking only whether a system predicts a MOS-aligned score, it asks whether a VLM can behave like a competent IQA agent that parses intent, reasons about degradations, chooses tools intelligently, and synthesizes intermediate evidence. In that sense it functions as a benchmark for agentic perception behavior, not merely end-to-end quality prediction (Zhu et al., 30 Sep 2025).

A common misconception follows from the acronym. Despite its name, AgenticIQA-Eval is distinct from “IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering” (Li et al., 2024). The latter concerns interactive question answering, where an LLM-based Evaluation Agent simulates a human user and then judges the dialogue. AgenticIQA-Eval instead concerns image quality assessment, is organized around planner/executor/summarizer roles, and is officially scored through MCQs rather than multi-turn interaction (Zhu et al., 30 Sep 2025, Li et al., 2024). The similarity in naming obscures a substantial difference in object of evaluation: interactive QA in one case, agentic image-quality reasoning in the other.

The benchmark also has explicit and implicit limitations. The paper notes error propagation, since the framework assumes reliable decomposition and tool selection; backbone dependence, because performance is bounded by the underlying VLM; and efficiency/scalability constraints, because sequential agentic reasoning is slower and more resource-intensive. In broader-impact discussion it also notes that VLM-based IQA may inherit biases from training data and amplify subjective quality judgments across demographic or cultural contexts (Zhu et al., 30 Sep 2025).

Benchmark-specific limitations are equally important. AgenticIQA-Eval is MCQ-based, so it does not directly score free-form explanation faithfulness. It does not explicitly score full trajectories or raw tool-execution correctness as a standalone metric. Robustness under major distribution shift is unclear, and the benchmark may depend on the particular distortion and tool ontology encoded by the framework. The paper therefore positions it as a meaningful step toward benchmarking agentic IQA ability, but not as a complete open-ended evaluation of all perceptual-agent behaviors (Zhu et al., 30 Sep 2025).